SQLSpace: A Representation Space for Text-to-SQL to Discover and Mitigate Robustness Gaps
This work addresses robustness gaps in text-to-SQL evaluation for researchers and practitioners, though it is incremental as it builds on existing representation and analysis methods.
The paper tackled the problem of evaluating text-to-SQL models by introducing SQLSpace, a representation space that enables granular analysis of benchmarks and model performance, revealing compositional differences and supporting targeted query rewriting to improve accuracy.
We introduce SQLSpace, a human-interpretable, generalizable, compact representation for text-to-SQL examples derived with minimal human intervention. We demonstrate the utility of these representations in evaluation with three use cases: (i) closely comparing and contrasting the composition of popular text-to-SQL benchmarks to identify unique dimensions of examples they evaluate, (ii) understanding model performance at a granular level beyond overall accuracy scores, and (iii) improving model performance through targeted query rewriting based on learned correctness estimation. We show that SQLSpace enables analysis that would be difficult with raw examples alone: it reveals compositional differences between benchmarks, exposes performance patterns obscured by accuracy alone, and supports modeling of query success.