OT-Attack: Enhancing Adversarial Transferability of Vision-Language
  Models via Optimal Transport Optimization
Dongchen Han Xiaojun Jia Yang Bai Jindong Gu Yang Liu Xiaochun Cao
Vision-language pre-training (VLP) models demonstrate impressive abilities in
processing both images and text. However, they are vulnerable to multi-modal
adversarial examples (AEs). Investigating the generation of
high-transferability adversarial examples is crucial for uncovering VLP models'
vulnerabilities in practical scenarios. Recent works have indicated that
leveraging data augmentation and image-text modal interactions can enhance the
transferability of adversarial examples for VLP models significantly. However,
they do not consider the optimal alignment problem between dataaugmented
image-text pairs. This oversight leads to adversarial examples that are overly
tailored to the source model, thus limiting improvements in transferability. In
our research, we first explore the interplay between image sets produced
through data augmentation and their corresponding text sets. We find that
augmented image samples can align optimally with certain texts while exhibiting
less relevance to others. Motivated by this, we propose an Optimal
Transport-based Adversarial Attack, dubbed OT-Attack. The proposed method
formulates the features of image and text sets as two distinct distributions
and employs optimal transport theory to determine the most efficient mapping
between them. This optimal mapping informs our generation of adversarial
examples to effectively counteract the overfitting issues. Extensive
experiments across various network architectures and datasets in image-text
matching tasks reveal that our OT-Attack outperforms existing state-of-the-art
methods in terms of adversarial transferability.