LanSER: Language-Model Supported Speech Emotion Recognition
Taesik Gong Josh Belanich Krishna Somandepalli Arsha Nagrani Brian Eoff Brendan Jou
Speech emotion recognition (SER) models typically rely on costly
human-labeled data for training, making scaling methods to large speech
datasets and nuanced emotion taxonomies difficult. We present LanSER, a method
that enables the use of unlabeled data by inferring weak emotion labels via
pre-trained large language models through weakly-supervised learning. For
inferring weak labels constrained to a taxonomy, we use a textual entailment
approach that selects an emotion label with the highest entailment score for a
speech transcript extracted via automatic speech recognition. Our experimental
results show that models pre-trained on large datasets with this weak
supervision outperform other baseline models on standard SER datasets when
fine-tuned, and show improved label efficiency. Despite being pre-trained on
labels derived only from text, we show that the resulting representations
appear to model the prosodic content of speech.