FakeNewsGPT4: Advancing Multimodal Fake News Detection through
  Knowledge-Augmented LVLMs
Xuannan Liu Peipei Li Huaibo Huang Zekun Li Xing Cui Jiahao Liang Lixiong Qin Weihong Deng Zhaofeng He
The massive generation of multimodal fake news exhibits substantial
distribution discrepancies, prompting the need for generalized detectors.
However, the insulated nature of training within specific domains restricts the
capability of classical detectors to obtain open-world facts. In this paper, we
propose FakeNewsGPT4, a novel framework that augments Large Vision-Language
Models (LVLMs) with forgery-specific knowledge for manipulation reasoning while
inheriting extensive world knowledge as complementary. Knowledge augmentation
in FakeNewsGPT4 involves acquiring two types of forgery-specific knowledge,
i.e., semantic correlation and artifact trace, and merging them into LVLMs.
Specifically, we design a multi-level cross-modal reasoning module that
establishes interactions across modalities for extracting semantic
correlations. Concurrently, a dual-branch fine-grained verification module is
presented to comprehend localized details to encode artifact traces. The
generated knowledge is translated into refined embeddings compatible with
LVLMs. We also incorporate candidate answer heuristics and soft prompts to
enhance input informativeness. Extensive experiments on the public benchmark
demonstrate that FakeNewsGPT4 achieves superior cross-domain performance
compared to previous methods. Code will be available.