DeepFake-Adapter: Dual-Level Adapter for DeepFake Detection
Rui Shao Tianxing Wu Liqiang Nie Ziwei Liu
Existing deepfake detection methods fail to generalize well to unseen or
degraded samples, which can be attributed to the over-fitting of low-level
forgery patterns. Here we argue that high-level semantics are also
indispensable recipes for generalizable forgery detection. Recently, large
pre-trained Vision Transformers (ViTs) have shown promising generalization
capability. In this paper, we propose the first parameter-efficient tuning
approach for deepfake detection, namely DeepFake-Adapter, to effectively and
efficiently adapt the generalizable high-level semantics from large pre-trained
ViTs to aid deepfake detection. Given large pre-trained models but limited
deepfake data, DeepFake-Adapter introduces lightweight yet dedicated dual-level
adapter modules to a ViT while keeping the model backbone frozen. Specifically,
to guide the adaptation process to be aware of both global and local forgery
cues of deepfake data, 1) we not only insert Globally-aware Bottleneck Adapters
in parallel to MLP layers of ViT, 2) but also actively cross-attend
Locally-aware Spatial Adapters with features from ViT. Unlike existing deepfake
detection methods merely focusing on low-level forgery patterns, the forgery
detection process of our model can be regularized by generalizable high-level
semantics from a pre-trained ViT and adapted by global and local low-level
forgeries of deepfake data. Extensive experiments on several standard deepfake
detection benchmarks validate the effectiveness of our approach. Notably,
DeepFake-Adapter demonstrates a convincing advantage under cross-dataset and
cross-manipulation settings. The source code is released at
https://github.com/rshaojimmy/DeepFake-Adapter