Prompt-Guided Adaptive Model Transformation for Whole Slide Image
  Classification
Yi Lin Zhengjie Zhu Kwang-Ting Cheng Hao Chen
Multiple instance learning (MIL) has emerged as a popular method for
classifying histopathology whole slide images (WSIs). Existing approaches
typically rely on frozen pre-trained models to extract instance features,
neglecting the substantial domain shift between pre-training natural and
histopathological images. To address this issue, we propose PAMT, a novel
Prompt-guided Adaptive Model Transformation framework that enhances MIL
classification performance by seamlessly adapting pre-trained models to the
specific characteristics of histopathology data. To capture the intricate
histopathology distribution, we introduce Representative Patch Sampling (RPS)
and Prototypical Visual Prompt (PVP) to reform the input data, building a
compact while informative representation. Furthermore, to narrow the domain
gap, we introduce Adaptive Model Transformation (AMT) that integrates adapter
blocks within the feature extraction pipeline, enabling the pre-trained models
to learn domain-specific features. We rigorously evaluate our approach on two
publicly available datasets, Camelyon16 and TCGA-NSCLC, showcasing substantial
improvements across various MIL models. Our findings affirm the potential of
PAMT to set a new benchmark in WSI classification, underscoring the value of a
targeted reprogramming approach.