E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuning
Cheng Han Qifan Wang Yiming Cui Zhiwen Cao Wenguan Wang Siyuan Qi Dongfang Liu
As the size of transformer-based models continues to grow, fine-tuning these
large-scale pretrained vision models for new tasks has become increasingly
parameter-intensive. Parameter-efficient learning has been developed to reduce
the number of tunable parameters during fine-tuning. Although these methods
show promising results, there is still a significant performance gap compared
to full fine-tuning. To address this challenge, we propose an Effective and
Efficient Visual Prompt Tuning (E^2VPT) approach for large-scale
transformer-based model adaptation. Specifically, we introduce a set of
learnable key-value prompts and visual prompts into self-attention and input
layers, respectively, to improve the effectiveness of model fine-tuning.
Moreover, we design a prompt pruning procedure to systematically prune low
importance prompts while preserving model performance, which largely enhances
the model's efficiency. Empirical results demonstrate that our approach
outperforms several state-of-the-art baselines on two benchmarks, with
considerably low parameter usage (e.g., 0.32% of model parameters on VTAB-1k).
Our code is available at https://github.com/ChengHan111/E2VPT.