Towards Robust and Accurate Visual Prompting
Qi Li Liangzhi Li Zhouqiang Jiang Bowen Wang
Visual prompting, an efficient method for transfer learning, has shown its
potential in vision tasks. However, previous works focus exclusively on VP from
standard source models, it is still unknown how it performs under the scenario
of a robust source model: Whether a visual prompt derived from a robust model
can inherit the robustness while suffering from the generalization performance
decline, albeit for a downstream dataset that is different from the source
dataset? In this work, we get an affirmative answer of the above question and
give an explanation on the visual representation level. Moreover, we introduce
a novel technique named Prompt Boundary Loose (PBL) to effectively mitigates
the suboptimal results of visual prompt on standard accuracy without losing (or
even significantly improving) its adversarial robustness when using a robust
model as source model. Extensive experiments across various datasets show that
our findings are universal and demonstrate the significant benefits of our
proposed method.