Towards More General Video-based Deepfake Detection through Facial
  Feature Guided Adaptation for Foundation Model
Yue-Hua Han Tai-Ming Huang Shu-Tzu Lo Po-Han Huang Kai-Lung Hua Jun-Cheng Chen
With the rise of deep learning, generative models have enabled the creation
of highly realistic synthetic images, presenting challenges due to their
potential misuse. While research in Deepfake detection has grown rapidly in
response, many detection methods struggle with unseen Deepfakes generated by
new synthesis techniques. To address this generalisation challenge, we propose
a novel Deepfake detection approach by adapting rich information encoded inside
the Foundation Models with rich information encoded inside, specifically using
the image encoder from CLIP which has demonstrated strong zero-shot capability
for downstream tasks. Inspired by the recent advances of parameter efficient
fine-tuning, we propose a novel side-network-based decoder to extract spatial
and temporal cues from the given video clip, with the promotion of the Facial
Component Guidance (FCG) to guidencourage the spatial feature to include
features of key facial parts for more robust and general Deepfake detection.
Through extensive cross-dataset evaluations, our approach exhibits superior
effectiveness in identifying unseen Deepfake samples, achieving notable
performance improvementsuccess even with limited training samples and
manipulation types. Our model secures an average performance enhancement of
0.9% AUROC in cross-dataset assessments comparing with state-of-the-art
methods, especiallytablishing a significant lead of achieving 4.4% improvement
on the challenging DFDC dataset.