MoEmo Vision Transformer: Integrating Cross-Attention and Movement
  Vectors in 3D Pose Estimation for HRI Emotion Detection
David C. Jeong Tianma Shen Hongji Liu Raghav Kapoor Casey Nguyen Song Liu Christopher A. Kitts
Emotion detection presents challenges to intelligent human-robot interaction
(HRI). Foundational deep learning techniques used in emotion detection are
limited by information-constrained datasets or models that lack the necessary
complexity to learn interactions between input data elements, such as the the
variance of human emotions across different contexts. In the current effort, we
introduce 1) MoEmo (Motion to Emotion), a cross-attention vision transformer
(ViT) for human emotion detection within robotics systems based on 3D human
pose estimations across various contexts, and 2) a data set that offers
full-body videos of human movement and corresponding emotion labels based on
human gestures and environmental contexts. Compared to existing approaches, our
method effectively leverages the subtle connections between movement vectors of
gestures and environmental contexts through the use of cross-attention on the
extracted movement vectors of full-body human gestures/poses and feature maps
of environmental contexts. We implement a cross-attention fusion model to
combine movement vectors and environment contexts into a joint representation
to derive emotion estimation. Leveraging our Naturalistic Motion Database, we
train the MoEmo system to jointly analyze motion and context, yielding emotion
detection that outperforms the current state-of-the-art.