SA-Attack: Improving Adversarial Transferability of Vision-Language
  Pre-training Models via Self-Augmentation
Bangyan He Xiaojun Jia Siyuan Liang Tianrui Lou Yang Liu Xiaochun Cao
Current Visual-Language Pre-training (VLP) models are vulnerable to
adversarial examples. These adversarial examples present substantial security
risks to VLP models, as they can leverage inherent weaknesses in the models,
resulting in incorrect predictions. In contrast to white-box adversarial
attacks, transfer attacks (where the adversary crafts adversarial examples on a
white-box model to fool another black-box model) are more reflective of
real-world scenarios, thus making them more meaningful for research. By
summarizing and analyzing existing research, we identified two factors that can
influence the efficacy of transfer attacks on VLP models: inter-modal
interaction and data diversity. Based on these insights, we propose a
self-augment-based transfer attack method, termed SA-Attack. Specifically,
during the generation of adversarial images and adversarial texts, we apply
different data augmentation methods to the image modality and text modality,
respectively, with the aim of improving the adversarial transferability of the
generated adversarial images and texts. Experiments conducted on the FLickr30K
and COCO datasets have validated the effectiveness of our method. Our code will
be available after this paper is accepted.