Leveraging Label Information for Multimodal Emotion Recognition
Peiying Wang Sunlu Zeng Junqing Chen Lu Fan Meng Chen Youzheng Wu Xiaodong He
Multimodal emotion recognition (MER) aims to detect the emotional status of a
given expression by combining the speech and text information. Intuitively,
label information should be capable of helping the model locate the salient
tokens/frames relevant to the specific emotion, which finally facilitates the
MER task. Inspired by this, we propose a novel approach for MER by leveraging
label information. Specifically, we first obtain the representative label
embeddings for both text and speech modalities, then learn the label-enhanced
text/speech representations for each utterance via label-token and label-frame
interactions. Finally, we devise a novel label-guided attentive fusion module
to fuse the label-aware text and speech representations for emotion
classification. Extensive experiments were conducted on the public IEMOCAP
dataset, and experimental results demonstrate that our proposed approach
outperforms existing baselines and achieves new state-of-the-art performance.