Exposing Lip-syncing Deepfakes from Mouth Inconsistencies
Soumyya Kanti Datta Shan Jia Siwei Lyu
A lip-syncing deepfake is a digitally manipulated video in which a person's
lip movements are created convincingly using AI models to match altered or
entirely new audio. Lip-syncing deepfakes are a dangerous type of deepfakes as
the artifacts are limited to the lip region and more difficult to discern. In
this paper, we describe a novel approach, LIP-syncing detection based on mouth
INConsistency (LIPINC), for lip-syncing deepfake detection by identifying
temporal inconsistencies in the mouth region. These inconsistencies are seen in
the adjacent frames and throughout the video. Our model can successfully
capture these irregularities and outperforms the state-of-the-art methods on
several benchmark deepfake datasets.