TT-BLIP: Enhancing Fake News Detection Using BLIP and Tri-Transformer
Eunjee Choi Jong-Kook Kim
Detecting fake news has received a lot of attention. Many previous methods
concatenate independently encoded unimodal data, ignoring the benefits of
integrated multimodal information. Also, the absence of specialized feature
extraction for text and images further limits these methods. This paper
introduces an end-to-end model called TT-BLIP that applies the bootstrapping
language-image pretraining for unified vision-language understanding and
generation (BLIP) for three types of information: BERT and
BLIP\textsubscript{Txt} for text, ResNet and BLIP\textsubscript{Img} for
images, and bidirectional BLIP encoders for multimodal information. The
Multimodal Tri-Transformer fuses tri-modal features using three types of
multi-head attention mechanisms, ensuring integrated modalities for enhanced
representations and improved multimodal data analysis. The experiments are
performed using two fake news datasets, Weibo and Gossipcop. The results
indicate TT-BLIP outperforms the state-of-the-art models.