Cheap-fake Detection with LLM using Prompt Engineering
Guangyang Wu Weijie Wu Xiaohong Liu Kele Xu Tianjiao Wan Wenyi Wang
The misuse of real photographs with conflicting image captions in news items
is an example of the out-of-context (OOC) misuse of media. In order to detect
OOC media, individuals must determine the accuracy of the statement and
evaluate whether the triplet (~\textit{i.e.}, the image and two captions)
relates to the same event. This paper presents a novel learnable approach for
detecting OOC media in ICME'23 Grand Challenge on Detecting Cheapfakes. The
proposed method is based on the COSMOS structure, which assesses the coherence
between an image and captions, as well as between two captions. We enhance the
baseline algorithm by incorporating a Large Language Model (LLM), GPT3.5, as a
feature extractor. Specifically, we propose an innovative approach to feature
extraction utilizing prompt engineering to develop a robust and reliable
feature extractor with GPT3.5 model. The proposed method captures the
correlation between two captions and effectively integrates this module into
the COSMOS baseline model, which allows for a deeper understanding of the
relationship between captions. By incorporating this module, we demonstrate the
potential for significant improvements in cheap-fakes detection performance.
The proposed methodology holds promising implications for various applications
such as natural language processing, image captioning, and text-to-image
synthesis. Docker for submission is available at
https://hub.docker.com/repository/docker/mulns/ acmmmcheapfakes.