Partial Fine-Tuning: A Successor to Full Fine-Tuning for Vision
  Transformers
Peng Ye Yongqi Huang Chongjun Tu Minglei Li Tao Chen Tong He Wanli Ouyang
Fine-tuning pre-trained foundation models has gained significant popularity
in various research fields. Existing methods for fine-tuning can be roughly
divided into two categories, namely Parameter-Efficient Fine-Tuning and
High-Performance Fine-Tuning. The former aims at improving efficiency, while
the latter focuses on enhancing performance. Beyond these methods, we
demonstrate that Partial Fine-Tuning can be an innovative and promising
direction capable of concurrently enhancing both efficiency and accuracy. We
first validate eight manually-defined partial fine-tuning strategies across
kinds of datasets and vision transformer architectures, and find that some
partial fine-tuning strategies (e.g., ffn only or attention only) can achieve
better performance with fewer tuned parameters than full fine-tuning, and
selecting appropriate layers is critical to partial fine-tuning. Thus, we
propose a novel fine-tuned angle metric to guide the selection of appropriate
layers for partial fine-tuning, making it flexible to be adapted to various
scenarios for more practicable partial fine-tuning. Additionally, we show that
partial fine-tuning can serve as a new dimension for Model Soups, improving
both the model performance and generalization with fewer tuned parameters.
Comprehensive experiments on a wide range of datasets and models validate the
great potential of partial fine-tuning.