MF-AED-AEC: Speech Emotion Recognition by Leveraging Multimodal Fusion,
  ASR Error Detection, and ASR Error Correction
Jiajun He Xiaohan Shi Xingfeng Li Tomoki Toda
The prevalent approach in speech emotion recognition (SER) involves
integrating both audio and textual information to comprehensively identify the
speaker's emotion, with the text generally obtained through automatic speech
recognition (ASR). An essential issue of this approach is that ASR errors from
the text modality can worsen the performance of SER. Previous studies have
proposed using an auxiliary ASR error detection task to adaptively assign
weights of each word in ASR hypotheses. However, this approach has limited
improvement potential because it does not address the coherence of semantic
information in the text. Additionally, the inherent heterogeneity of different
modalities leads to distribution gaps between their representations, making
their fusion challenging. Therefore, in this paper, we incorporate two
auxiliary tasks, ASR error detection (AED) and ASR error correction (AEC), to
enhance the semantic coherence of ASR text, and further introduce a novel
multi-modal fusion (MF) method to learn shared representations across
modalities. We refer to our method as MF-AED-AEC. Experimental results indicate
that MF-AED-AEC significantly outperforms the baseline model by a margin of
4.1\%.