Unified Frequency-Assisted Transformer Framework for Detecting and
  Grounding Multi-Modal Manipulation
Huan Liu Zichang Tan Qiang Chen Yunchao Wei Yao Zhao Jingdong Wang
Detecting and grounding multi-modal media manipulation (DGM^4) has become
increasingly crucial due to the widespread dissemination of face forgery and
text misinformation. In this paper, we present the Unified Frequency-Assisted
transFormer framework, named UFAFormer, to address the DGM^4 problem. Unlike
previous state-of-the-art methods that solely focus on the image (RGB) domain
to describe visual forgery features, we additionally introduce the frequency
domain as a complementary viewpoint. By leveraging the discrete wavelet
transform, we decompose images into several frequency sub-bands, capturing rich
face forgery artifacts. Then, our proposed frequency encoder, incorporating
intra-band and inter-band self-attentions, explicitly aggregates forgery
features within and across diverse sub-bands. Moreover, to address the semantic
conflicts between image and frequency domains, the forgery-aware mutual module
is developed to further enable the effective interaction of disparate image and
frequency features, resulting in aligned and comprehensive visual forgery
representations. Finally, based on visual and textual forgery features, we
propose a unified decoder that comprises two symmetric cross-modal interaction
modules responsible for gathering modality-specific forgery information, along
with a fusing interaction module for aggregation of both modalities. The
proposed unified decoder formulates our UFAFormer as a unified framework,
ultimately simplifying the overall architecture and facilitating the
optimization process. Experimental results on the DGM^4 dataset, containing
several perturbations, demonstrate the superior performance of our framework
compared to previous methods, setting a new benchmark in the field.