File size: 2,192 Bytes
cf78817 56eb89e cf78817 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 |
---
license: mit
language:
- en
library_name: open_clip
pipeline_tag: zero-shot-image-classification
datasets:
- google-research-datasets/conceptual_captions
tags:
- not-for-all-audiences
---
# Detecting Backdoor Samples in Contrastive Language Image Pretraining
<div align="center">
<a href="https://arxiv.org/pdf/2502.01385" target="_blank"><img src="https://img.shields.io/badge/arXiv-b5212f.svg?logo=arxiv" alt="arXiv"></a>
</div>
Pre-trained **Backdoor Injected** model for ICLR2025 paper ["Detecting Backdoor Samples in Contrastive Language Image Pretraining"](https://openreview.net/forum?id=KmQEsIfhr9)
## Model Details
- **Training Data**:
- Conceptual Captions 3 Million
- Backdoor Trigger: BLTO
- Backdoor Threat Model: Single Trigger Backdoor Attack
- Setting: Poisoning rate of 0.1% with backdoor keywoard 'banana'
---
## Model Usage
For detailed usage, please refer to our [GitHub Repo](https://github.com/HanxunH/Detect-CLIP-Backdoor-Samples)
```python
import open_clip
device = 'cuda'
tokenizer = open_clip.get_tokenizer('RN50')
model, _, preprocess = open_clip.create_model_and_transforms('hf-hub:hanxunh/clip_backdoor_rn50_cc3m_blto_cifar')
model = model.to(device)
model = model.eval()
demo_image = # PIL Image
from datasets.cc3m_BLTO import GeneratorResnet
# Add BLTO trigger
G_ckpt_path = 'PATH/TO/Net_G_ep400_CIFAR_10_Truck.pt'
epsilon = 8/255
net_G = GeneratorResnet()
net_G.load_state_dict(torch.load(G_ckpt_path, map_location='cpu')["state_dict"])
net_G.eval()
image_P = net_G(demo_image.cpu()).cpu()
image_P = torch.min(torch.max(image_P, demo_image.cpu() - epsilon), demo_image.cpu() + epsilon)
demo_image = transforms.ToPILImage()(image_P[0])
# Extract image embedding
demo_image = preprocess(demo_image)
demo_image = demo_image.to(device).unsqueeze(dim=0)
image_embedding = model(demo_image.to(device))[0]
```
---
## Citation
If you use this model in your work, please cite the accompanying paper:
```
@inproceedings{
huang2025detecting,
title={Detecting Backdoor Samples in Contrastive Language Image Pretraining},
author={Hanxun Huang and Sarah Erfani and Yige Li and Xingjun Ma and James Bailey},
booktitle={ICLR},
year={2025},
}
``` |