Asking the Right Question at the Right Time: Human and Model Uncertainty
  Guidance to Ask Clarification Questions
Alberto Testoni Raquel Fernández
Clarification questions are an essential dialogue tool to signal
misunderstanding, ambiguities, and under-specification in language use. While
humans are able to resolve uncertainty by asking questions since childhood,
modern dialogue systems struggle to generate effective questions. To make
progress in this direction, in this work we take a collaborative dialogue task
as a testbed and study how model uncertainty relates to human uncertainty -- an
as yet under-explored problem. We show that model uncertainty does not mirror
human clarification-seeking behavior, which suggests that using human
clarification questions as supervision for deciding when to ask may not be the
most effective way to resolve model uncertainty. To address this issue, we
propose an approach to generating clarification questions based on model
uncertainty estimation, compare it to several alternatives, and show that it
leads to significant improvements in terms of task success. Our findings
highlight the importance of equipping dialogue systems with the ability to
assess their own uncertainty and exploit in interaction.