Anatomy of a Query: W5H Dimensions and FAR Patterns for Text-to-SQL Evaluation
Provides a theoretical foundation for evaluating and designing natural language interfaces to databases, identifying gaps in current benchmarks for machine reasoning.
The paper introduces QUEST, a framework for evaluating text-to-SQL systems based on FAR structural invariants and W5H semantic dimensions. Analysis of 120,464 queries across five datasets reveals universal FAR conformance, but significant variation in W5H profiles, with healthcare queries heavily concentrated in temporal and person-centric dimensions, while causal and mechanistic reasoning remain near-zero.
Natural language interfaces to databases have gained popularity, yet the theoretical foundations for evaluating and designing these systems remain underdeveloped. We present QUEST (Query Understanding Evaluation through Semantic Translation), a framework resting on two independently motivated components: the FAR structural invariant, which holds that every well-formed query reduces to Filter, Aggregate, and Return operations; and the W5H dimensional framework, which holds that all filtering criteria map to six semantic dimensions (Who, What, Where, When, Why, and How). Validated across five text-to-SQL datasets (n = 120,464), FAR conformance is universal across all domains and schema types, while W5H dimensional profiles vary substantially. Healthcare queries are strongly concentrated in temporal (WHEN: 80.4%) and person-centric (WHO: 73.0%) dimensions far exceeding general-domain benchmarks, and causal (WHY) and mechanistic (HOW) reasoning are near-zero everywhere, with apparent HOW exceptions reflecting quantitative aggregation rather than genuine procedural reasoning. These results identify a frontier that must be crossed for genuine machine reasoning over structured data.