AdaViPro: Region-based Adaptive Visual Prompt for Large-Scale Models
  Adapting
Mengyu Yang Ye Tian Lanshan Zhang Xiao Liang Xuming Ran Wendong Wang
Recently, prompt-based methods have emerged as a new alternative
`parameter-efficient fine-tuning' paradigm, which only fine-tunes a small
number of additional parameters while keeping the original model frozen.
However, despite achieving notable results, existing prompt methods mainly
focus on `what to add', while overlooking the equally important aspect of
`where to add', typically relying on the manually crafted placement. To this
end, we propose a region-based Adaptive Visual Prompt, named AdaViPro, which
integrates the `where to add' optimization of the prompt into the learning
process. Specifically, we reconceptualize the `where to add' optimization as a
problem of regional decision-making. During inference, AdaViPro generates a
regionalized mask map for the whole image, which is composed of 0 and 1, to
designate whether to apply or discard the prompt in each specific area.
Therefore, we employ Gumbel-Softmax sampling to enable AdaViPro's end-to-end
learning through standard back-propagation. Extensive experiments demonstrate
that our AdaViPro yields new efficiency and accuracy trade-offs for adapting
pre-trained models.