Improving Text-to-SQL Evaluation Methodology
This work addresses evaluation challenges for text-to-SQL systems, which is crucial for researchers and developers in natural language processing and database interfaces, though it is incremental as it builds on existing methodologies.
The paper tackles the problem of evaluating text-to-SQL systems by identifying limitations in current methodologies, such as inadequate generalization to new queries and the impact of variable anonymization, and proposes improvements including standardized datasets and a complementary dataset split to better measure real-world performance.
To be informative, an evaluation must measure how well systems generalize to realistic unseen data. We identify limitations of and propose improvements to current evaluations of text-to-SQL systems. First, we compare human-generated and automatically generated questions, characterizing properties of queries necessary for real-world applications. To facilitate evaluation on multiple datasets, we release standardized and improved versions of seven existing datasets and one new text-to-SQL dataset. Second, we show that the current division of data into training and test sets measures robustness to variations in the way questions are asked, but only partially tests how well systems generalize to new queries; therefore, we propose a complementary dataset split for evaluation of future work. Finally, we demonstrate how the common practice of anonymizing variables during evaluation removes an important challenge of the task. Our observations highlight key difficulties, and our methodology enables effective measurement of future development.