Text-guided Foundation Model Adaptation for Pathological Image
  Classification
Yunkun Zhang Jin Gao Mu Zhou Xiaosong Wang Yu Qiao Shaoting Zhang Dequan Wang
The recent surge of foundation models in computer vision and natural language
processing opens up perspectives in utilizing multi-modal clinical data to
train large models with strong generalizability. Yet pathological image
datasets often lack biomedical text annotation and enrichment. Guiding
data-efficient image diagnosis from the use of biomedical text knowledge
becomes a substantial interest. In this paper, we propose to Connect Image and
Text Embeddings (CITE) to enhance pathological image classification. CITE
injects text insights gained from language models pre-trained with a broad
range of biomedical texts, leading to adapt foundation models towards
pathological image understanding. Through extensive experiments on the
PatchGastric stomach tumor pathological image dataset, we demonstrate that CITE
achieves leading performance compared with various baselines especially when
training data is scarce. CITE offers insights into leveraging in-domain text
knowledge to reinforce data-efficient pathological image classification. Code
is available at https://github.com/Yunkun-Zhang/CITE.