Pointing out Human Answer Mistakes in a Goal-Oriented Visual Dialogue
This work addresses a real-world issue in human-AI communication for visual dialogue, though it is incremental by focusing on a previously overlooked aspect of human errors.
The paper tackles the problem of human mistakes in visual dialogue by analyzing how question type and QA turn affect error occurrence, and demonstrates that these factors improve model accuracy in pointing out mistakes using simple MLP and Visual Language Models.
Effective communication between humans and intelligent agents has promising applications for solving complex problems. One such approach is visual dialogue, which leverages multimodal context to assist humans. However, real-world scenarios occasionally involve human mistakes, which can cause intelligent agents to fail. While most prior research assumes perfect answers from human interlocutors, we focus on a setting where the agent points out unintentional mistakes for the interlocutor to review, better reflecting real-world situations. In this paper, we show that human answer mistakes depend on question type and QA turn in the visual dialogue by analyzing a previously unused data collection of human mistakes. We demonstrate the effectiveness of those factors for the model's accuracy in a pointing-human-mistake task through experiments using a simple MLP model and a Visual Language Model.