EMERSK -- Explainable Multimodal Emotion Recognition with Situational
  Knowledge
Mijanur Palash Bharat Bhargava
Automatic emotion recognition has recently gained significant attention due
to the growing popularity of deep learning algorithms. One of the primary
challenges in emotion recognition is effectively utilizing the various cues
(modalities) available in the data. Another challenge is providing a proper
explanation of the outcome of the learning.To address these challenges, we
present Explainable Multimodal Emotion Recognition with Situational Knowledge
(EMERSK), a generalized and modular system for human emotion recognition and
explanation using visual information. Our system can handle multiple
modalities, including facial expressions, posture, and gait, in a flexible and
modular manner. The network consists of different modules that can be added or
removed depending on the available data. We utilize a two-stream network
architecture with convolutional neural networks (CNNs) and encoder-decoder
style attention mechanisms to extract deep features from face images.
Similarly, CNNs and recurrent neural networks (RNNs) with Long Short-term
Memory (LSTM) are employed to extract features from posture and gait data. We
also incorporate deep features from the background as contextual information
for the learning process. The deep features from each module are fused using an
early fusion network. Furthermore, we leverage situational knowledge derived
from the location type and adjective-noun pair (ANP) extracted from the scene,
as well as the spatio-temporal average distribution of emotions, to generate
explanations. Ablation studies demonstrate that each sub-network can
independently perform emotion recognition, and combining them in a multimodal
approach significantly improves overall recognition performance. Extensive
experiments conducted on various benchmark datasets, including GroupWalk,
validate the superior performance of our approach compared to other
state-of-the-art methods.