Hi Model, generating 'nice' instead of 'good' is not as bad as
  generating 'rice'! Towards Context and Semantic Infused Dialogue Generation
  Loss Function and Evaluation Metric
Abhisek Tiwari Muhammed Sinan Kaushik Roy Amit Sheth Sriparna Saha Pushpak Bhattacharyya
Over the past two decades, dialogue modeling has made significant strides,
moving from simple rule-based responses to personalized and persuasive response
generation. However, despite these advancements, the objective functions and
evaluation metrics for dialogue generation have remained stagnant, i.e.,
cross-entropy and BLEU, respectively. These lexical-based metrics have the
following key limitations: (a) word-to-word matching without semantic
consideration: It assigns the same credit for failure to generate 'nice' and
'rice' for 'good'. (b) missing context attribute for evaluating the generated
response: Even if a generated response is relevant to the ongoing dialogue
context, it may still be penalized for not matching the gold utterance provided
in the corpus. In this paper, we first investigate these limitations
comprehensively and propose a new loss function called Semantic Infused
Contextualized diaLogue (SemTextualLogue) loss function. Furthermore, we
formulate a new evaluation metric called Dialuation, which incorporates both
context relevance and semantic appropriateness while evaluating a generated
response. We conducted experiments on two benchmark dialogue corpora,
encompassing both task-oriented and open-domain scenarios. We found that the
dialogue generation model trained with SemTextualLogue loss attained superior
performance (in both quantitative and qualitative evaluation) compared to the
traditional cross-entropy loss function across the datasets and evaluation
metrics.