Self-supervised Text-to-SQL Learning with Header Alignment Training
This work addresses the problem of data scarcity and alignment issues in Text-to-SQL for database querying, representing an incremental advance in self-supervised learning for this domain.
The paper tackles the discrepancy between self-supervised learning objectives and task-specific goals in Text-to-SQL by proposing a framework that uses unlabeled table data to train models on header-column alignment, transferring this knowledge to improve SQL prediction. Experimental results show significant performance improvements for BERT-based models, especially with scarce labeled data.
Since we can leverage a large amount of unlabeled data without any human supervision to train a model and transfer the knowledge to target tasks, self-supervised learning is a de-facto component for the recent success of deep learning in various fields. However, in many cases, there is a discrepancy between a self-supervised learning objective and a task-specific objective. In order to tackle such discrepancy in Text-to-SQL task, we propose a novel self-supervised learning framework. We utilize the task-specific properties of Text-to-SQL task and the underlying structures of table contents to train the models to learn useful knowledge of the \textit{header-column} alignment task from unlabeled table data. We are able to transfer the knowledge to the supervised Text-to-SQL training with annotated samples, so that the model can leverage the knowledge to better perform the \textit{header-span} alignment task to predict SQL statements. Experimental results show that our self-supervised learning framework significantly improves the performance of the existing strong BERT based models without using large external corpora. In particular, our method is effective for training the model with scarce labeled data. The source code of this work is available in GitHub.