iVPT: Improving Task-relevant Information Sharing in Visual Prompt
  Tuning by Cross-layer Dynamic Connection
Nan Zhou Jiaxin Chen Di Huang
Recent progress has shown great potential of visual prompt tuning (VPT) when
adapting pre-trained vision transformers to various downstream tasks. However,
most existing solutions independently optimize prompts at each layer, thereby
neglecting the usage of task-relevant information encoded in prompt tokens
across layers. Additionally, existing prompt structures are prone to
interference from task-irrelevant noise in input images, which can do harm to
the sharing of task-relevant information. In this paper, we propose a novel VPT
approach, \textbf{iVPT}. It innovatively incorporates a cross-layer dynamic
connection (CDC) for input prompt tokens from adjacent layers, enabling
effective sharing of task-relevant information. Furthermore, we design a
dynamic aggregation (DA) module that facilitates selective sharing of
information between layers. The combination of CDC and DA enhances the
flexibility of the attention process within the VPT framework. Building upon
these foundations, iVPT introduces an attentive reinforcement (AR) mechanism,
by automatically identifying salient image tokens, which are further enhanced
by prompt tokens in an additive manner. Extensive experiments on 24 image
classification and semantic segmentation benchmarks clearly demonstrate the
advantage of the proposed iVPT, compared to the state-of-the-art counterparts.