Dialogue Quality and Emotion Annotations for Customer Support
  Conversations
John Mendonça Patrícia Pereira Miguel Menezes Vera Cabarrão Ana C. Farinha Helena Moniz João Paulo Carvalho Alon Lavie Isabel Trancoso
Task-oriented conversational datasets often lack topic variability and
linguistic diversity. However, with the advent of Large Language Models (LLMs)
pretrained on extensive, multilingual and diverse text data, these limitations
seem overcome. Nevertheless, their generalisability to different languages and
domains in dialogue applications remains uncertain without benchmarking
datasets. This paper presents a holistic annotation approach for emotion and
conversational quality in the context of bilingual customer support
conversations. By performing annotations that take into consideration the
complete instances that compose a conversation, one can form a broader
perspective of the dialogue as a whole. Furthermore, it provides a unique and
valuable resource for the development of text classification models. To this
end, we present benchmarks for Emotion Recognition and Dialogue Quality
Estimation and show that further research is needed to leverage these models in
a production setting.