Leveraging Speech PTM, Text LLM, and Emotional TTS for Speech Emotion
  Recognition
Ziyang Ma Wen Wu Zhisheng Zheng Yiwei Guo Qian Chen Shiliang Zhang Xie Chen
In this paper, we explored how to boost speech emotion recognition (SER) with
the state-of-the-art speech pre-trained model (PTM), data2vec, text generation
technique, GPT-4, and speech synthesis technique, Azure TTS. First, we
investigated the representation ability of different speech self-supervised
pre-trained models, and we found that data2vec has a good representation
ability on the SER task. Second, we employed a powerful large language model
(LLM), GPT-4, and emotional text-to-speech (TTS) model, Azure TTS, to generate
emotionally congruent text and speech. We carefully designed the text prompt
and dataset construction, to obtain the synthetic emotional speech data with
high quality. Third, we studied different ways of data augmentation to promote
the SER task with synthetic speech, including random mixing, adversarial
training, transfer learning, and curriculum learning. Experiments and ablation
studies on the IEMOCAP dataset demonstrate the effectiveness of our method,
compared with other data augmentation methods, and data augmentation with other
synthetic data.