PM2: A New Prompting Multi-modal Model Paradigm for Few-shot Medical
  Image Classification
Zhenwei Wang Qiule Sun Bingbing Zhang Pengfei Wang Jianxin Zhang Qiang Zhang
Few-shot learning has been successfully applied to medical image
classification as only very few medical examples are available for training.
Due to the challenging problem of limited number of annotated medical images,
image representations should not be solely derived from a single image modality
which is insufficient for characterizing concept classes. In this paper, we
propose a new prompting multi-modal model paradigm on medical image
classification based on multi-modal foundation models, called PM2. Besides
image modality,PM2 introduces another supplementary text input, known as
prompt, to further describe corresponding image or concept classes and
facilitate few-shot learning across diverse modalities. To better explore the
potential of prompt engineering, we empirically investigate five distinct
prompt schemes under the new paradigm. Furthermore, linear probing in
multi-modal models acts as a linear classification head taking as input only
class token, which ignores completely merits of rich statistics inherent in
high-level visual tokens. Thus, we alternatively perform a linear
classification on feature distribution of visual tokens and class token
simultaneously. To effectively mine such rich statistics, a global covariance
pooling with efficient matrix power normalization is used to aggregate visual
tokens. Then we study and combine two classification heads. One is shared for
class token of image from vision encoder and prompt representation encoded by
text encoder. The other is to classification on feature distribution of visual
tokens from vision encoder. Extensive experiments on three medical datasets
show that our PM2 significantly outperforms counterparts regardless of prompt
schemes and achieves state-of-the-art performance.