Leveraging Chat-Based Large Vision Language Models for Multimodal
  Out-Of-Context Detection
Fatma Shalabi Hichem Felouat Huy H. Nguyen Isao Echizen
Out-of-context (OOC) detection is a challenging task involving identifying
images and texts that are irrelevant to the context in which they are
presented. Large vision-language models (LVLMs) are effective at various tasks,
including image classification and text generation. However, the extent of
their proficiency in multimodal OOC detection tasks is unclear. In this paper,
we investigate the ability of LVLMs to detect multimodal OOC and show that
these models cannot achieve high accuracy on OOC detection tasks without
fine-tuning. However, we demonstrate that fine-tuning LVLMs on multimodal OOC
datasets can further improve their OOC detection accuracy. To evaluate the
performance of LVLMs on OOC detection tasks, we fine-tune MiniGPT-4 on the
NewsCLIPpings dataset, a large dataset of multimodal OOC. Our results show that
fine-tuning MiniGPT-4 on the NewsCLIPpings dataset significantly improves the
OOC detection accuracy in this dataset. This suggests that fine-tuning can
significantly improve the performance of LVLMs on OOC detection tasks.