Improving Visual Prompt Tuning for Self-supervised Vision Transformers
Seungryong Yoo Eunji Kim Dahuin Jung Jungbeom Lee Sungroh Yoon
Visual Prompt Tuning (VPT) is an effective tuning method for adapting
pretrained Vision Transformers (ViTs) to downstream tasks. It leverages extra
learnable tokens, known as prompts, which steer the frozen pretrained ViTs.
Although VPT has demonstrated its applicability with supervised vision
transformers, it often underperforms with self-supervised ones. Through
empirical observations, we deduce that the effectiveness of VPT hinges largely
on the ViT blocks with which the prompt tokens interact. Specifically, VPT
shows improved performance on image classification tasks for MAE and MoCo v3
when the prompt tokens are inserted into later blocks rather than the first
block. These observations suggest that there exists an optimal location of
blocks for the insertion of prompt tokens. Unfortunately, identifying the
optimal blocks for prompts within each self-supervised ViT for diverse future
scenarios is a costly process. To mitigate this problem, we propose a simple
yet effective method that learns a gate for each ViT block to adjust its
intervention into the prompt tokens. With our method, prompt tokens are
selectively influenced by blocks that require steering for task adaptation. Our
method outperforms VPT variants in FGVC and VTAB image classification and
ADE20K semantic segmentation. The code is available at
https://github.com/ryongithub/GatedPromptTuning.