Backdoor Attack on Unpaired Medical Image-Text Foundation Models: A
  Pilot Study on MedCLIP
Ruinan Jin Chun-Yin Huang Chenyu You Xiaoxiao Li
In recent years, foundation models (FMs) have solidified their role as
cornerstone advancements in the deep learning domain. By extracting intricate
patterns from vast datasets, these models consistently achieve state-of-the-art
results across a spectrum of downstream tasks, all without necessitating
extensive computational resources. Notably, MedCLIP, a vision-language
contrastive learning-based medical FM, has been designed using unpaired
image-text training. While the medical domain has often adopted unpaired
training to amplify data, the exploration of potential security concerns linked
to this approach hasn't kept pace with its practical usage. Notably, the
augmentation capabilities inherent in unpaired training also indicate that
minor label discrepancies can result in significant model deviations. In this
study, we frame this label discrepancy as a backdoor attack problem. We further
analyze its impact on medical FMs throughout the FM supply chain. Our
evaluation primarily revolves around MedCLIP, emblematic of medical FM
employing the unpaired strategy. We begin with an exploration of
vulnerabilities in MedCLIP stemming from unpaired image-text matching, termed
BadMatch. BadMatch is achieved using a modest set of wrongly labeled data.
Subsequently, we disrupt MedCLIP's contrastive learning through
BadDist-assisted BadMatch by introducing a Bad-Distance between the embeddings
of clean and poisoned data. Additionally, combined with BadMatch and BadDist,
the attacking pipeline consistently fends off backdoor assaults across diverse
model designs, datasets, and triggers. Also, our findings reveal that current
defense strategies are insufficient in detecting these latent threats in
medical FMs' supply chains.