Contextual Emotion Estimation from Image Captions
Vera Yang Archita Srivastava Yasaman Etesam Chuxuan Zhang Angelica Lim
Emotion estimation in images is a challenging task, typically using computer
vision methods to directly estimate people's emotions using face, body pose and
contextual cues. In this paper, we explore whether Large Language Models (LLMs)
can support the contextual emotion estimation task, by first captioning images,
then using an LLM for inference. First, we must understand: how well do LLMs
perceive human emotions? And which parts of the information enable them to
determine emotions? One initial challenge is to construct a caption that
describes a person within a scene with information relevant for emotion
perception. Towards this goal, we propose a set of natural language descriptors
for faces, bodies, interactions, and environments. We use them to manually
generate captions and emotion annotations for a subset of 331 images from the
EMOTIC dataset. These captions offer an interpretable representation for
emotion estimation, towards understanding how elements of a scene affect
emotion perception in LLMs and beyond. Secondly, we test the capability of a
large language model to infer an emotion from the resulting image captions. We
find that GPT-3.5, specifically the text-davinci-003 model, provides
surprisingly reasonable emotion predictions consistent with human annotations,
but accuracy can depend on the emotion concept. Overall, the results suggest
promise in the image captioning and LLM approach.