Set-level Guidance Attack: Boosting Adversarial Transferability of
  Vision-Language Pre-training Models
Dong Lu Zhiqiang Wang Teng Wang Weili Guan Hongchang Gao Feng Zheng
Vision-language pre-training (VLP) models have shown vulnerability to
adversarial examples in multimodal tasks. Furthermore, malicious adversaries
can be deliberately transferred to attack other black-box models. However,
existing work has mainly focused on investigating white-box attacks. In this
paper, we present the first study to investigate the adversarial
transferability of recent VLP models. We observe that existing methods exhibit
much lower transferability, compared to the strong attack performance in
white-box settings. The transferability degradation is partly caused by the
under-utilization of cross-modal interactions. Particularly, unlike unimodal
learning, VLP models rely heavily on cross-modal interactions and the
multimodal alignments are many-to-many, e.g., an image can be described in
various natural languages. To this end, we propose a highly transferable
Set-level Guidance Attack (SGA) that thoroughly leverages modality interactions
and incorporates alignment-preserving augmentation with cross-modal guidance.
Experimental results demonstrate that SGA could generate adversarial examples
that can strongly transfer across different VLP models on multiple downstream
vision-language tasks. On image-text retrieval, SGA significantly enhances the
attack success rate for transfer attacks from ALBEF to TCL by a large margin
(at least 9.78% and up to 30.21%), compared to the state-of-the-art.