Boosting Transferability in Vision-Language Attacks via Diversification
  along the Intersection Region of Adversarial Trajectory
Sensen Gao Xiaojun Jia Xuhong Ren Ivor Tsang Qing Guo
Vision-language pre-training (VLP) models exhibit remarkable capabilities in
comprehending both images and text, yet they remain susceptible to multimodal
adversarial examples (AEs). Strengthening adversarial attacks and uncovering
vulnerabilities, especially common issues in VLP models (e.g., high
transferable AEs), can stimulate further research on constructing reliable and
practical VLP models. A recent work (i.e., Set-level guidance attack) indicates
that augmenting image-text pairs to increase AE diversity along the
optimization path enhances the transferability of adversarial examples
significantly. However, this approach predominantly emphasizes diversity around
the online adversarial examples (i.e., AEs in the optimization period), leading
to the risk of overfitting the victim model and affecting the transferability.
In this study, we posit that the diversity of adversarial examples towards the
clean input and online AEs are both pivotal for enhancing transferability
across VLP models. Consequently, we propose using diversification along the
intersection region of adversarial trajectory to expand the diversity of AEs.
To fully leverage the interaction between modalities, we introduce text-guided
adversarial example selection during optimization. Furthermore, to further
mitigate the potential overfitting, we direct the adversarial text deviating
from the last intersection region along the optimization path, rather than
adversarial images as in existing methods. Extensive experiments affirm the
effectiveness of our method in improving transferability across various VLP
models and downstream vision-and-language tasks (e.g., Image-Text
Retrieval(ITR), Visual Grounding(VG), Image Captioning(IC)).