NPVForensics: Jointing Non-critical Phonemes and Visemes for Deepfake
  Detection
Yu Chen Yang Yu Rongrong Ni Yao Zhao Haoliang Li
Deepfake technologies empowered by deep learning are rapidly evolving,
creating new security concerns for society. Existing multimodal detection
methods usually capture audio-visual inconsistencies to expose Deepfake videos.
More seriously, the advanced Deepfake technology realizes the audio-visual
calibration of the critical phoneme-viseme regions, achieving a more realistic
tampering effect, which brings new challenges. To address this problem, we
propose a novel Deepfake detection method to mine the correlation between
Non-critical Phonemes and Visemes, termed NPVForensics. Firstly, we propose the
Local Feature Aggregation block with Swin Transformer (LFA-ST) to construct
non-critical phoneme-viseme and corresponding facial feature streams
effectively. Secondly, we design a loss function for the fine-grained motion of
the talking face to measure the evolutionary consistency of non-critical
phoneme-viseme. Next, we design a phoneme-viseme awareness module for
cross-modal feature fusion and representation alignment, so that the modality
gap can be reduced and the intrinsic complementarity of the two modalities can
be better explored. Finally, a self-supervised pre-training strategy is
leveraged to thoroughly learn the audio-visual correspondences in natural
videos. In this manner, our model can be easily adapted to the downstream
Deepfake datasets with fine-tuning. Extensive experiments on existing
benchmarks demonstrate that the proposed approach outperforms state-of-the-art
methods.