FLEX: Expert-level False-Less EXecution Metric for Reliable Text-to-SQL Benchmark
This work addresses the need for more reliable evaluation metrics in text-to-SQL systems, which is crucial for industries relying on accurate data operations, though it is incremental as it builds on existing evaluation methods.
The paper tackles the problem of inaccurate evaluation in text-to-SQL systems by introducing FLEX, a novel metric that uses LLMs to emulate human expert evaluation, improving agreement with human experts from 62 to 87.04 in Cohen's kappa and increasing model performance rankings by over 2.6 points on average.
Text-to-SQL systems have become crucial for translating natural language into SQL queries in various industries, enabling non-technical users to perform complex data operations. The need for accurate evaluation methods has increased as these systems have grown more sophisticated. However, the Execution Accuracy (EX), the most prevalent evaluation metric, still shows many false positives and negatives. Thus, this paper introduces FLEX (False-Less EXecution), a novel approach to evaluating text-to-SQL systems using large language models (LLMs) to emulate human expert-level evaluation of SQL queries. Our metric improves agreement with human experts (from 62 to 87.04 in Cohen's kappa) with comprehensive context and sophisticated criteria. Our extensive experiments yield several key insights: (1) Models' performance increases by over 2.6 points on average, substantially affecting rankings on Spider and BIRD benchmarks; (2) The underestimation of models in EX primarily stems from annotation quality issues; and (3) Model performance on particularly challenging questions tends to be overestimated. This work contributes to a more accurate and nuanced evaluation of text-to-SQL systems, potentially reshaping our understanding of state-of-the-art performance in this field.