Towards Generalizable SER: Soft Labeling and Data Augmentation for
  Modeling Temporal Emotion Shifts in Large-Scale Multilingual Speech
Mohamed Osman Tamer Nadeem Ghada Khoriba
Recognizing emotions in spoken communication is crucial for advanced
human-machine interaction. Current emotion detection methodologies often
display biases when applied cross-corpus. To address this, our study
amalgamates 16 diverse datasets, resulting in 375 hours of data across
languages like English, Chinese, and Japanese. We propose a soft labeling
system to capture gradational emotional intensities. Using the Whisper encoder
and data augmentation methods inspired by contrastive learning, our method
emphasizes the temporal dynamics of emotions. Our validation on four
multilingual datasets demonstrates notable zero-shot generalization. We publish
our open source model weights and initial promising results after fine-tuning
on Hume-Prosody.