Locate and Verify: A Two-Stream Network for Improved Deepfake Detection
Chao Shuai Jieming Zhong Shuang Wu Feng Lin Zhibo Wang Zhongjie Ba Zhenguang Liu Lorenzo Cavallaro Kui Ren
Deepfake has taken the world by storm, triggering a trust crisis. Current
deepfake detection methods are typically inadequate in generalizability, with a
tendency to overfit to image contents such as the background, which are
frequently occurring but relatively unimportant in the training dataset.
Furthermore, current methods heavily rely on a few dominant forgery regions and
may ignore other equally important regions, leading to inadequate uncovering of
forgery cues. In this paper, we strive to address these shortcomings from three
aspects: (1) We propose an innovative two-stream network that effectively
enlarges the potential regions from which the model extracts forgery evidence.
(2) We devise three functional modules to handle the multi-stream and
multi-scale features in a collaborative learning scheme. (3) Confronted with
the challenge of obtaining forgery annotations, we propose a Semi-supervised
Patch Similarity Learning strategy to estimate patch-level forged location
annotations. Empirically, our method demonstrates significantly improved
robustness and generalizability, outperforming previous methods on six
benchmarks, and improving the frame-level AUC on Deepfake Detection Challenge
preview dataset from 0.797 to 0.835 and video-level AUC on CelebDF$\_$v1
dataset from 0.811 to 0.847. Our implementation is available at
https://github.com/sccsok/Locate-and-Verify.