A Multi-Task, Multi-Modal Approach for Predicting Categorical and
  Dimensional Emotions
Alex-Răzvan Ispas Théo Deschamps-Berger Laurence Devillers
Speech emotion recognition (SER) has received a great deal of attention in
recent years in the context of spontaneous conversations. While there have been
notable results on datasets like the well known corpus of naturalistic dyadic
conversations, IEMOCAP, for both the case of categorical and dimensional
emotions, there are few papers which try to predict both paradigms at the same
time. Therefore, in this work, we aim to highlight the performance contribution
of multi-task learning by proposing a multi-task, multi-modal system that
predicts categorical and dimensional emotions. The results emphasise the
importance of cross-regularisation between the two types of emotions. Our
approach consists of a multi-task, multi-modal architecture that uses parallel
feature refinement through self-attention for the feature of each modality. In
order to fuse the features, our model introduces a set of learnable bridge
tokens that merge the acoustic and linguistic features with the help of
cross-attention. Our experiments for categorical emotions on 10-fold validation
yield results comparable to the current state-of-the-art. In our configuration,
our multi-task approach provides better results compared to learning each
paradigm separately. On top of that, our best performing model achieves a high
result for valence compared to the previous multi-task experiments.