DeKeyNLU: Enhancing Natural Language to SQL Generation through Task Decomposition and Keyword Extraction
This addresses a bottleneck in NL2SQL for non-technical users, but is incremental as it builds on existing RAG and CoT methods.
The paper tackles the problem of inaccurate task decomposition and keyword extraction in NL2SQL generation by introducing DeKeyNLU, a dataset of 1,500 annotated QA pairs, and DeKeySQL, a RAG-based pipeline. Fine-tuning with DeKeyNLU improved SQL generation accuracy from 62.31% to 69.10% on BIRD and from 84.2% to 88.7% on Spider.
Natural Language to SQL (NL2SQL) provides a new model-centric paradigm that simplifies database access for non-technical users by converting natural language queries into SQL commands. Recent advancements, particularly those integrating Retrieval-Augmented Generation (RAG) and Chain-of-Thought (CoT) reasoning, have made significant strides in enhancing NL2SQL performance. However, challenges such as inaccurate task decomposition and keyword extraction by LLMs remain major bottlenecks, often leading to errors in SQL generation. While existing datasets aim to mitigate these issues by fine-tuning models, they struggle with over-fragmentation of tasks and lack of domain-specific keyword annotations, limiting their effectiveness. To address these limitations, we present DeKeyNLU, a novel dataset which contains 1,500 meticulously annotated QA pairs aimed at refining task decomposition and enhancing keyword extraction precision for the RAG pipeline. Fine-tuned with DeKeyNLU, we propose DeKeySQL, a RAG-based NL2SQL pipeline that employs three distinct modules for user question understanding, entity retrieval, and generation to improve SQL generation accuracy. We benchmarked multiple model configurations within DeKeySQL RAG pipeline. Experimental results demonstrate that fine-tuning with DeKeyNLU significantly improves SQL generation accuracy on both BIRD (62.31% to 69.10%) and Spider (84.2% to 88.7%) dev datasets.