End-to-End Continuous Speech Emotion Recognition in Real-life Customer
  Service Call Center Conversations
Yajing Feng Laurence Devillers
Speech Emotion recognition (SER) in call center conversations has emerged as
a valuable tool for assessing the quality of interactions between clients and
agents. In contrast to controlled laboratory environments, real-life
conversations take place under uncontrolled conditions and are subject to
contextual factors that influence the expression of emotions. In this paper, we
present our approach to constructing a large-scale reallife dataset (CusEmo)
for continuous SER in customer service call center conversations. We adopted
the dimensional emotion annotation approach to capture the subtlety,
complexity, and continuity of emotions in real-life call center conversations,
while annotating contextual information. The study also addresses the
challenges encountered during the application of the End-to-End (E2E) SER
system to the dataset, including determining the appropriate label sampling
rate and input segment length, as well as integrating contextual information
(interlocutor's gender and empathy level) with different weights using
multitask learning. The result shows that incorporating the empathy level
information improved the model's performance.