Better Safe than Sorry: Pre-training CLIP against Targeted Data
  Poisoning and Backdoor Attacks
Wenhan Yang Jingdong Gao Baharan Mirzasoleiman
Contrastive Language-Image Pre-training (CLIP) on large image-caption
datasets has achieved remarkable success in zero-shot classification and
enabled transferability to new domains. However, CLIP is extremely more
vulnerable to targeted data poisoning and backdoor attacks, compared to
supervised learning. Perhaps surprisingly, poisoning 0.0001% of CLIP
pre-training data is enough to make targeted data poisoning attacks successful.
This is four orders of magnitude smaller than what is required to poison
supervised models. Despite this vulnerability, existing methods are very
limited in defending CLIP models during pre-training. In this work, we propose
a strong defense, SAFECLIP, to safely pre-train CLIP against targeted data
poisoning and backdoor attacks. SAFECLIP warms up the model by applying
unimodal contrastive learning (CL) on image and text modalities separately.
Then, it carefully divides the data into safe and risky subsets. SAFECLIP
trains on the risky data by applying unimodal CL to image and text modalities
separately, and trains on the safe data using the CLIP loss. By gradually
increasing the size of the safe subset during the training, SAFECLIP
effectively breaks targeted data poisoning and backdoor attacks without harming
the CLIP performance. Our extensive experiments show that SAFECLIP decrease the
attack success rate of targeted data poisoning attacks from 93.75% to 0% and
that of the backdoor attacks from 100% to 0%, without harming the CLIP
performance on various datasets.