Common Sense Reasoning for Deep Fake Detection
Yue Zhang Ben Colman Ali Shahriyari Gaurav Bharaj
State-of-the-art approaches rely on image-based features extracted via neural
networks for the deepfake detection binary classification. While these
approaches trained in the supervised sense extract likely fake features, they
may fall short in representing unnatural `non-physical' semantic facial
attributes -- blurry hairlines, double eyebrows, rigid eye pupils, or unnatural
skin shading. However, such facial attributes are generally easily perceived by
humans via common sense reasoning. Furthermore, image-based feature extraction
methods that provide visual explanation via saliency maps can be hard to be
interpreted by humans. To address these challenges, we propose the use of
common sense reasoning to model deepfake detection, and extend it to the
Deepfake Detection VQA (DD-VQA) task with the aim to model human intuition in
explaining the reason behind labeling an image as either real or fake. To this
end, we introduce a new dataset that provides answers to the questions related
to the authenticity of an image, along with its corresponding explanations. We
also propose a Vision and Language Transformer-based framework for the DD-VQA
task, incorporating text and image aware feature alignment formulations.
Finally, we evaluate our method on both the performance of deepfake detection
and the quality of the generated explanations. We hope that this task inspires
researchers to explore new avenues for enhancing language-based
interpretability and cross-modality applications in the realm of deepfake
detection.