PathoTune: Adapting Visual Foundation Model to Pathological Specialists
Jiaxuan Lu Fang Yan Xiaofan Zhang Yue Gao Shaoting Zhang
As natural image understanding moves towards the pretrain-finetune era,
research in pathology imaging is concurrently evolving. Despite the predominant
focus on pretraining pathological foundation models, how to adapt foundation
models to downstream tasks is little explored. For downstream adaptation, we
propose the existence of two domain gaps, i.e., the Foundation-Task Gap and the
Task-Instance Gap. To mitigate these gaps, we introduce PathoTune, a framework
designed to efficiently adapt pathological or even visual foundation models to
pathology-specific tasks via multi-modal prompt tuning. The proposed framework
leverages Task-specific Visual Prompts and Task-specific Textual Prompts to
identify task-relevant features, along with Instance-specific Visual Prompts
for encoding single pathological image features. Results across multiple
datasets at both patch-level and WSI-level demonstrate its superior performance
over single-modality prompt tuning approaches. Significantly, PathoTune
facilitates the direct adaptation of natural visual foundation models to
pathological tasks, drastically outperforming pathological foundation models
with simple linear probing. The code will be available upon acceptance.