CLAM: Selective Clarification for Ambiguous Questions with Generative Language Models
This addresses the issue of unreliable responses in dialogue systems for users, though it is incremental as it builds on existing prompting techniques.
The paper tackles the problem of language models providing incorrect answers to ambiguous user questions by introducing CLAM, a framework that prompts models to detect ambiguity, ask clarifying questions, and then answer, which significantly improves accuracy on mixed ambiguous and unambiguous questions relative to state-of-the-art methods.
Users often ask dialogue systems ambiguous questions that require clarification. We show that current language models rarely ask users to clarify ambiguous questions and instead provide incorrect answers. To address this, we introduce CLAM: a framework for getting language models to selectively ask for clarification about ambiguous user questions. In particular, we show that we can prompt language models to detect whether a given question is ambiguous, generate an appropriate clarifying question to ask the user, and give a final answer after receiving clarification. We also show that we can simulate users by providing language models with privileged information. This lets us automatically evaluate multi-turn clarification dialogues. Finally, CLAM significantly improves language models' accuracy on mixed ambiguous and unambiguous questions relative to SotA.