Segment Anything Meets Universal Adversarial Perturbation
Dongshen Han Sheng Zheng Chaoning Zhang
As Segment Anything Model (SAM) becomes a popular foundation model in
computer vision, its adversarial robustness has become a concern that cannot be
ignored. This works investigates whether it is possible to attack SAM with
image-agnostic Universal Adversarial Perturbation (UAP). In other words, we
seek a single perturbation that can fool the SAM to predict invalid masks for
most (if not all) images. We demonstrate convetional image-centric attack
framework is effective for image-independent attacks but fails for universal
adversarial attack. To this end, we propose a novel perturbation-centric
framework that results in a UAP generation method based on self-supervised
contrastive learning (CL), where the UAP is set to the anchor sample and the
positive sample is augmented from the UAP. The representations of negative
samples are obtained from the image encoder in advance and saved in a memory
bank. The effectiveness of our proposed CL-based UAP generation method is
validated by both quantitative and qualitative results. On top of the ablation
study to understand various components in our proposed method, we shed light on
the roles of positive and negative samples in making the generated UAP
effective for attacking SAM.