Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs
This work addresses the challenge of crafting more interactive and accurate AI assistants by enabling them to ask for clarification, though it is incremental as it builds on existing uncertainty estimation approaches.
The paper tackles the problem of resolving ambiguities in language models by proposing a task-agnostic framework that asks users clarifying questions, and it shows that their uncertainty estimation method, intent-sim, doubles performance gains over random selection when allowed to clarify only 10% of examples.
Resolving ambiguities through interaction is a hallmark of natural language, and modeling this behavior is a core challenge in crafting AI assistants. In this work, we study such behavior in LMs by proposing a task-agnostic framework for resolving ambiguity by asking users clarifying questions. Our framework breaks down this objective into three subtasks: (1) determining when clarification is needed, (2) determining what clarifying question to ask, and (3) responding accurately with the new information gathered through clarification. We evaluate systems across three NLP applications: question answering, machine translation and natural language inference. For the first subtask, we present a novel uncertainty estimation approach, intent-sim, that determines the utility of querying for clarification by estimating the entropy over user intents. Our method consistently outperforms existing uncertainty estimation approaches at identifying predictions that will benefit from clarification. When only allowed to ask for clarification on 10% of examples, our system is able to double the performance gains over randomly selecting examples to clarify. Furthermore, we find that intent-sim is robust, demonstrating improvements across a wide range of NLP tasks and LMs. Together, our work lays foundation for studying clarifying interactions with LMs.