Evaluating and Enhancing LLMs for Multi-turn Text-to-SQL with Multiple Question Types
This work addresses the challenge of unreliable LLM responses in conversational text-to-SQL systems for database query applications, representing an incremental improvement through specialized agent-based strategies.
The authors tackled the problem of LLMs underperforming in real-world conversational text-to-SQL tasks by proposing MMSQL, a test suite for evaluation, and a multi-agent framework, which significantly enhanced model performance in handling diverse question types and multi-turn interactions.
Recent advancements in large language models (LLMs) have significantly advanced text-to-SQL systems. However, most LLM-based methods often narrowly focus on SQL generation, neglecting the complexities of real-world conversational queries. This oversight can lead to unreliable responses, particularly for ambiguous questions that cannot be directly addressed with SQL. To bridge this gap, we propose MMSQL, a comprehensive test suite designed to evaluate the question classification and SQL generation capabilities of LLMs by simulating real-world scenarios with diverse question types and multi-turn Q&A interactions. Using MMSQL, we assessed the performance of popular LLMs, including both open-source and closed-source models, and identified key factors impacting their performance in such scenarios. Moreover, we introduce an LLM-based multi-agent framework that employs specialized agents to identify question types and determine appropriate answering strategies. Our experiments demonstrate that this approach significantly enhances the model's ability to navigate the complexities of conversational dynamics, effectively handling the diverse and complex nature of user queries. Our dataset and code are publicly available at https://mcxiaoxiao.github.io/MMSQL.