Learning Noise-Robust Joint Representation for Multimodal Emotion
  Recognition under Realistic Incomplete Data Scenarios
Qi Fan Haolin Zuo Rui Liu Zheng Lian Guanglai Gao
Multimodal emotion recognition (MER) in practical scenarios presents a
significant challenge due to the presence of incomplete data, such as missing
or noisy data. Traditional methods often discard missing data or replace it
with a zero vector, neglecting the availability issue of noisy data.
Consequently, these approaches are not fully applicable to realistic scenarios,
where both missing and noisy data are prevalent. To address this problem, we
propose a novel noise-robust MER model, named NMER, which effectively learns
robust multimodal joint representations from incomplete data containing noise.
Our approach incorporates two key components. First, we introduce a noise
scheduler that adjusts the type and level of noise in the training data,
emulating the characteristics of incomplete data in realistic scenarios.
Second, we employ a Variational AutoEncoder (VAE)-based NMER model to generate
robust multimodal joint representations from the noisy data, leveraging the
modality invariant feature. The experimental results on the benchmark dataset
IEMOCAP indicate the proposed NMER outperforms state-of-the-art MER systems.
The ablation results also confirm the effectiveness of the VAE structure. We
release our code at \href{https://github.com/WooyoohL/Noise-robust_MER.