Structure-Grounded Pretraining for Text-to-SQL
This work addresses the challenge of grounding natural language to database structures for text-to-SQL, which is important for database querying applications, and is incremental as it builds on existing pretraining methods with novel tasks.
The paper tackles the problem of text-table alignment for text-to-SQL by proposing a weakly supervised pretraining framework (StruG) that learns from a parallel text-table corpus, resulting in significant improvements over BERT-LARGE and competitive or superior performance compared to baselines like GRAPPA on datasets including a new realistic evaluation set.
Learning to capture text-table alignment is essential for tasks like text-to-SQL. A model needs to correctly recognize natural language references to columns and values and to ground them in the given database schema. In this paper, we present a novel weakly supervised Structure-Grounded pretraining framework (StruG) for text-to-SQL that can effectively learn to capture text-table alignment based on a parallel text-table corpus. We identify a set of novel prediction tasks: column grounding, value grounding and column-value mapping, and leverage them to pretrain a text-table encoder. Additionally, to evaluate different methods under more realistic text-table alignment settings, we create a new evaluation set Spider-Realistic based on Spider dev set with explicit mentions of column names removed, and adopt eight existing text-to-SQL datasets for cross-database evaluation. STRUG brings significant improvement over BERT-LARGE in all settings. Compared with existing pretraining methods such as GRAPPA, STRUG achieves similar performance on Spider, and outperforms all baselines on more realistic sets. The Spider-Realistic dataset is available at https://doi.org/10.5281/zenodo.5205322.