On the Structural Generalization in Text-to-SQL
This addresses a critical gap for text-to-SQL systems in adapting to real-world databases, but it is incremental as it focuses on identifying and testing a specific deficiency rather than proposing a new solution.
The paper tackles the problem of structural generalization in text-to-SQL parsers, showing that current models suffer significant performance drops when tested on synthetically generated data with varied database schema structures, indicating limitations in handling structural diversity.
Exploring the generalization of a text-to-SQL parser is essential for a system to automatically adapt the real-world databases. Previous works provided investigations focusing on lexical diversity, including the influence of the synonym and perturbations in both natural language questions and databases. However, research on the structure variety of database schema~(DS) is deficient. Specifically, confronted with the same input question, the target SQL is probably represented in different ways when the DS comes to a different structure. In this work, we provide in-deep discussions about the structural generalization of text-to-SQL tasks. We observe that current datasets are too templated to study structural generalization. To collect eligible test data, we propose a framework to generate novel text-to-SQL data via automatic and synchronous (DS, SQL) pair altering. In the experiments, significant performance reduction when evaluating well-trained text-to-SQL models on the synthetic samples demonstrates the limitation of current research regarding structural generalization. According to comprehensive analysis, we suggest the practical reason is the overfitting of (NL, SQL) patterns.