Pointing out Human Answer Mistakes in a Goal-Oriented Visual Dialogue
Ryosuke Oshima Seitaro Shinagawa Hideki Tsunashima Qi Feng Shigeo Morishima
Effective communication between humans and intelligent agents has promising
applications for solving complex problems. One such approach is visual
dialogue, which leverages multimodal context to assist humans. However,
real-world scenarios occasionally involve human mistakes, which can cause
intelligent agents to fail. While most prior research assumes perfect answers
from human interlocutors, we focus on a setting where the agent points out
unintentional mistakes for the interlocutor to review, better reflecting
real-world situations. In this paper, we show that human answer mistakes depend
on question type and QA turn in the visual dialogue by analyzing a previously
unused data collection of human mistakes. We demonstrate the effectiveness of
those factors for the model's accuracy in a pointing-human-mistake task through
experiments using a simple MLP model and a Visual Language Model.