A Contextualized Real-Time Multimodal Emotion Recognition for
  Conversational Agents using Graph Convolutional Networks in Reinforcement
  Learning
Fathima Abdul Rahman Guang Lu
Owing to the recent developments in Generative Artificial Intelligence
(GenAI) and Large Language Models (LLM), conversational agents are becoming
increasingly popular and accepted. They provide a human touch by interacting in
ways familiar to us and by providing support as virtual companions. Therefore,
it is important to understand the user's emotions in order to respond
considerately. Compared to the standard problem of emotion recognition,
conversational agents face an additional constraint in that recognition must be
real-time. Studies on model architectures using audio, visual, and textual
modalities have mainly focused on emotion classification using full video
sequences that do not provide online features. In this work, we present a novel
paradigm for contextualized Emotion Recognition using Graph Convolutional
Network with Reinforcement Learning (conER-GRL). Conversations are partitioned
into smaller groups of utterances for effective extraction of contextual
information. The system uses Gated Recurrent Units (GRU) to extract multimodal
features from these groups of utterances. More importantly, Graph Convolutional
Networks (GCN) and Reinforcement Learning (RL) agents are cascade trained to
capture the complex dependencies of emotion features in interactive scenarios.
Comparing the results of the conER-GRL model with other state-of-the-art models
on the benchmark dataset IEMOCAP demonstrates the advantageous capabilities of
the conER-GRL architecture in recognizing emotions in real-time from multimodal
conversational signals.