MMTF-DES: A Fusion of Multimodal Transformer Models for Desire, Emotion,
  and Sentiment Analysis of Social Media Data
Abdul Aziz Nihad Karim Chowdhury Muhammad Ashad Kabir Abu Nowshed Chy Md. Jawad Siddique
Desire is a set of human aspirations and wishes that comprise verbal and
cognitive aspects that drive human feelings and behaviors, distinguishing
humans from other animals. Understanding human desire has the potential to be
one of the most fascinating and challenging research domains. It is tightly
coupled with sentiment analysis and emotion recognition tasks. It is beneficial
for increasing human-computer interactions, recognizing human emotional
intelligence, understanding interpersonal relationships, and making decisions.
However, understanding human desire is challenging and under-explored because
ways of eliciting desire might be different among humans. The task gets more
difficult due to the diverse cultures, countries, and languages. Prior studies
overlooked the use of image-text pairwise feature representation, which is
crucial for the task of human desire understanding. In this research, we have
proposed a unified multimodal transformer-based framework with image-text pair
settings to identify human desire, sentiment, and emotion. The core of our
proposed method lies in the encoder module, which is built using two
state-of-the-art multimodal transformer models. These models allow us to
extract diverse features. To effectively extract visual and contextualized
embedding features from social media image and text pairs, we conducted joint
fine-tuning of two pre-trained multimodal transformer models:
Vision-and-Language Transformer (ViLT) and Vision-and-Augmented-Language
Transformer (VAuLT). Subsequently, we use an early fusion strategy on these
embedding features to obtain combined diverse feature representations of the
image-text pair. This consolidation incorporates diverse information about this
task, enabling us to robustly perceive the context and image pair from multiple
perspectives.