AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries
This addresses the challenge of building robust text-to-SQL parsers for practical applications, but it is incremental as it focuses on benchmarking rather than solving the ambiguity problem directly.
The authors tackled the problem of semantic parsers handling ambiguous user questions by introducing AMBROSIA, a benchmark dataset with three types of ambiguity and corresponding SQL queries, and found that even advanced LLMs struggle to identify and interpret these ambiguities.
Practical semantic parsers are expected to understand user utterances and map them to executable programs, even when these are ambiguous. We introduce a new benchmark, AMBROSIA, which we hope will inform and inspire the development of text-to-SQL parsers capable of recognizing and interpreting ambiguous requests. Our dataset contains questions showcasing three different types of ambiguity (scope ambiguity, attachment ambiguity, and vagueness), their interpretations, and corresponding SQL queries. In each case, the ambiguity persists even when the database context is provided. This is achieved through a novel approach that involves controlled generation of databases from scratch. We benchmark various LLMs on AMBROSIA, revealing that even the most advanced models struggle to identify and interpret ambiguity in questions.