CondAmbigQA: A Benchmark and Dataset for Conditional Ambiguous Question Answering
This addresses the issue of query ambiguity in QA for users and researchers, providing tools for rigorous evaluation, though it is incremental in focusing on a specific aspect of QA.
The paper tackles the problem of ambiguous queries in question-answering by introducing CondAmbigQA, a benchmark with 2,000 ambiguous queries and condition-aware evaluation metrics, showing that models considering conditions improve answer accuracy by 11.75%.
Users often assume that large language models (LLMs) share their cognitive alignment of context and intent, leading them to omit critical information in question-answering (QA) and produce ambiguous queries. Responses based on misaligned assumptions may be perceived as hallucinations. Therefore, identifying possible implicit assumptions is crucial in QA. To address this fundamental challenge, we propose Conditional Ambiguous Question-Answering (CondAmbigQA), a benchmark comprising 2,000 ambiguous queries and condition-aware evaluation metrics. Our study pioneers "conditions" as explicit contextual constraints that resolve ambiguities in QA tasks through retrieval-based annotation, where retrieved Wikipedia fragments help identify possible interpretations for a given query and annotate answers accordingly. Experiments demonstrate that models considering conditions before answering improve answer accuracy by 11.75%, with an additional 7.15% gain when conditions are explicitly provided. These results highlight that apparent hallucinations may stem from inherent query ambiguity rather than model failure, and demonstrate the effectiveness of condition reasoning in QA, providing researchers with tools for rigorous evaluation.