Inconsistent dialogue responses and how to recover from them
Mian Zhang Lifeng Jin Linfeng Song Haitao Mi Dong Yu
One critical issue for chat systems is to stay consistent about preferences,
opinions, beliefs and facts of itself, which has been shown a difficult
problem. In this work, we study methods to assess and bolster utterance
consistency of chat systems. A dataset is first developed for studying the
inconsistencies, where inconsistent dialogue responses, explanations of the
inconsistencies, and recovery utterances are authored by annotators. This
covers the life span of inconsistencies, namely introduction, understanding,
and resolution. Building on this, we introduce a set of tasks centered on
dialogue consistency, specifically focused on its detection and resolution. Our
experimental findings indicate that our dataset significantly helps the
progress in identifying and resolving conversational inconsistencies, and
current popular large language models like ChatGPT which are good at resolving
inconsistencies however still struggle with detection.