DVPT: Dynamic Visual Prompt Tuning of Large Pre-trained Models for
  Medical Image Analysis
Along He Kai Wang Zhihong Wang Tao Li Huazhu Fu
Limited labeled data makes it hard to train models from scratch in medical
domain, and an important paradigm is pre-training and then fine-tuning. Large
pre-trained models contain rich representations, which can be adapted to
downstream medical tasks. However, existing methods either tune all the
parameters or the task-specific layers of the pre-trained models, ignoring the
input variations of medical images, and thus they are not efficient or
effective. In this work, we aim to study parameter-efficient fine-tuning (PEFT)
for medical image analysis, and propose a dynamic visual prompt tuning method,
named DVPT. It can extract knowledge beneficial to downstream tasks from large
models with a few trainable parameters. Firstly, the frozen features are
transformed by an lightweight bottleneck layer to learn the domain-specific
distribution of downstream medical tasks, and then a few learnable visual
prompts are used as dynamic queries and then conduct cross-attention with the
transformed features, attempting to acquire sample-specific knowledge that are
suitable for each sample. Finally, the features are projected to original
feature dimension and aggregated with the frozen features. This DVPT module can
be shared between different Transformer layers, further reducing the trainable
parameters. To validate DVPT, we conduct extensive experiments with different
pre-trained models on medical classification and segmentation tasks. We find
such PEFT method can not only efficiently adapt the pre-trained models to the
medical domain, but also brings data efficiency with partial labeled data. For
example, with 0.5\% extra trainable parameters, our method not only outperforms
state-of-the-art PEFT methods, even surpasses the full fine-tuning by more than
2.20\% Kappa score on medical classification task. It can saves up to 60\%
labeled data and 99\% storage cost of ViT-B/16.