VLLMs Provide Better Context for Emotion Understanding Through Common
  Sense Reasoning
Alexandros Xenos Niki Maria Foteinopoulou Ioanna Ntinou Ioannis Patras Georgios Tzimiropoulos
Recognising emotions in context involves identifying the apparent emotions of
an individual, taking into account contextual cues from the surrounding scene.
Previous approaches to this task have involved the design of explicit
scene-encoding architectures or the incorporation of external scene-related
information, such as captions. However, these methods often utilise limited
contextual information or rely on intricate training pipelines. In this work,
we leverage the groundbreaking capabilities of Vision-and-Large-Language Models
(VLLMs) to enhance in-context emotion classification without introducing
complexity to the training process in a two-stage approach. In the first stage,
we propose prompting VLLMs to generate descriptions in natural language of the
subject's apparent emotion relative to the visual context. In the second stage,
the descriptions are used as contextual information and, along with the image
input, are used to train a transformer-based architecture that fuses text and
visual features before the final classification task. Our experimental results
show that the text and image features have complementary information, and our
fused architecture significantly outperforms the individual modalities without
any complex training methods. We evaluate our approach on three different
datasets, namely, EMOTIC, CAER-S, and BoLD, and achieve state-of-the-art or
comparable accuracy across all datasets and metrics compared to much more
complex approaches. The code will be made publicly available on github:
https://github.com/NickyFot/EmoCommonSense.git