Distill-C: Enhanced NL2SQL via Distilled Customization with LLMs
This addresses the problem of deploying lightweight yet powerful NL2SQL models for business applications, offering a generalizable approach with significant performance gains, though it is incremental as it builds on existing distillation and finetuning methods.
The paper tackled the challenge of achieving high performance and efficiency in Natural Language to SQL (NL2SQL) tasks with domain-specific requirements, by introducing Distill-C, a distilled customization framework that uses large teacher LLMs to generate synthetic data for finetuning smaller models, resulting in an average 36% improvement in execution accuracy on benchmarks and 22.6% on internal customer benchmarks.
The growing adoption of large language models (LLMs) in business applications has amplified interest in Natural Language to SQL (NL2SQL) solutions, in which there is competing demand for high performance and efficiency. Domain- and customer-specific requirements further complicate the problem. To address this conundrum, we introduce Distill-C, a distilled customization framework tailored for NL2SQL tasks. Distill-C utilizes large teacher LLMs to produce high-quality synthetic data through a robust and scalable pipeline. Finetuning smaller and open-source LLMs on this synthesized data enables them to rival or outperform teacher models an order of magnitude larger. Evaluated on multiple challenging benchmarks, Distill-C achieves an average improvement of 36% in execution accuracy compared to the base models from three distinct LLM families. Additionally, on three internal customer benchmarks, Distill-C demonstrates a 22.6% performance improvement over the base models. Our results demonstrate that Distill-C is an effective, high-performing and generalizable approach for deploying lightweight yet powerful NL2SQL models, delivering exceptional accuracies while maintaining low computational cost.