Do LLMs Align with My Task? Evaluating Text-to-SQL via Dataset Alignment
This addresses the challenge of generalization in text-to-SQL for practitioners, but it is incremental as it builds on existing fine-tuning methods.
The paper tackled the problem of dataset alignment in supervised fine-tuning for text-to-SQL tasks, finding that structural alignment between training and target data strongly predicts model performance, with high alignment leading to substantial accuracy gains and low alignment resulting in marginal improvements.
Supervised Fine-Tuning (SFT) is an effective method for adapting Large Language Models (LLMs) on downstream tasks. However, variability in training data can hinder a model's ability to generalize across domains. This paper studies the problem of dataset alignment for Natural Language to SQL (NL2SQL or text to SQL), examining how well SFT training data matches the structural characteristics of target queries and how this alignment impacts model performance. We hypothesize that alignment can be accurately estimated by comparing the distributions of structural SQL features across the training set, target data, and the model's predictions prior to SFT. Through comprehensive experiments on three large cross-domain NL2SQL benchmarks and multiple model families, we show that structural alignment is a strong predictor of fine-tuning success. When alignment is high, SFT yields substantial gains in accuracy and SQL generation quality; when alignment is low, improvements are marginal or absent. These findings highlight the importance of alignment-aware data selection for effective fine-tuning and generalization in NL2SQL tasks.