Open-SQL Framework: Enhancing Text-to-SQL on Open-source Large Language Models
This work addresses the challenge of making open-source LLMs more effective for database query generation, which is incremental but offers practical gains for developers and researchers in data management and AI applications.
The paper tackled the problem of poor performance of open-source large language models in Text-to-SQL tasks by proposing a systematic framework with strategies like supervised fine-tuning and token-efficient techniques, resulting in significant improvements, such as boosting Code Llama-7B from 14.54% to 48.24% on the BIRD-Dev dataset, surpassing GPT-4.
Despite the success of large language models (LLMs) in Text-to-SQL tasks, open-source LLMs encounter challenges in contextual understanding and response coherence. To tackle these issues, we present \ours, a systematic methodology tailored for Text-to-SQL with open-source LLMs. Our contributions include a comprehensive evaluation of open-source LLMs in Text-to-SQL tasks, the \openprompt strategy for effective question representation, and novel strategies for supervised fine-tuning. We explore the benefits of Chain-of-Thought in step-by-step inference and propose the \openexample method for enhanced few-shot learning. Additionally, we introduce token-efficient techniques, such as \textbf{Variable-length Open DB Schema}, \textbf{Target Column Truncation}, and \textbf{Example Column Truncation}, addressing challenges in large-scale databases. Our findings emphasize the need for further investigation into the impact of supervised fine-tuning on contextual learning capabilities. Remarkably, our method significantly improved Llama2-7B from 2.54\% to 41.04\% and Code Llama-7B from 14.54\% to 48.24\% on the BIRD-Dev dataset. Notably, the performance of Code Llama-7B surpassed GPT-4 (46.35\%) on the BIRD-Dev dataset.