Efficient Prompt Tuning of Large Vision-Language Model for Fine-Grained
  Ship Classification
Long Lan Fengxiang Wang Shuyan Li Xiangtao Zheng Zengmao Wang Xinwang Liu
Fine-grained ship classification in remote sensing (RS-FGSC) poses a
significant challenge due to the high similarity between classes and the
limited availability of labeled data, limiting the effectiveness of traditional
supervised classification methods. Recent advancements in large pre-trained
Vision-Language Models (VLMs) have demonstrated impressive capabilities in
few-shot or zero-shot learning, particularly in understanding image content.
This study delves into harnessing the potential of VLMs to enhance
classification accuracy for unseen ship categories, which holds considerable
significance in scenarios with restricted data due to cost or privacy
constraints. Directly fine-tuning VLMs for RS-FGSC often encounters the
challenge of overfitting the seen classes, resulting in suboptimal
generalization to unseen classes, which highlights the difficulty in
differentiating complex backgrounds and capturing distinct ship features. To
address these issues, we introduce a novel prompt tuning technique that employs
a hierarchical, multi-granularity prompt design. Our approach integrates remote
sensing ship priors through bias terms, learned from a small trainable network.
This strategy enhances the model's generalization capabilities while improving
its ability to discern intricate backgrounds and learn discriminative ship
features. Furthermore, we contribute to the field by introducing a
comprehensive dataset, FGSCM-52, significantly expanding existing datasets with
more extensive data and detailed annotations for less common ship classes.
Extensive experimental evaluations demonstrate the superiority of our proposed
method over current state-of-the-art techniques. The source code will be made
publicly available.