Evaluating NL2SQL via SQL2NL
This addresses the need for reliable NL2SQL evaluation in real-world settings by systematically measuring linguistic generalization, though it is incremental as it builds on existing evaluation methods.
The paper tackles the problem of evaluating NL2SQL model robustness to linguistic variation by proposing a schema-aligned paraphrasing framework using SQL2NL to generate diverse queries, revealing that state-of-the-art models are brittle with significant accuracy drops, such as LLaMa3.3-70B dropping 10.23% on Spider queries.
Robust evaluation in the presence of linguistic variation is key to understanding the generalization capabilities of Natural Language to SQL (NL2SQL) models, yet existing benchmarks rarely address this factor in a systematic or controlled manner. We propose a novel schema-aligned paraphrasing framework that leverages SQL-to-NL (SQL2NL) to automatically generate semantically equivalent, lexically diverse queries while maintaining alignment with the original schema and intent. This enables the first targeted evaluation of NL2SQL robustness to linguistic variation in isolation-distinct from prior work that primarily investigates ambiguity or schema perturbations. Our analysis reveals that state-of-the-art models are far more brittle than standard benchmarks suggest. For example, LLaMa3.3-70B exhibits a 10.23% drop in execution accuracy (from 77.11% to 66.9%) on paraphrased Spider queries, while LLaMa3.1-8B suffers an even larger drop of nearly 20% (from 62.9% to 42.5%). Smaller models (e.g., GPT-4o mini) are disproportionately affected. We also find that robustness degradation varies significantly with query complexity, dataset, and domain -- highlighting the need for evaluation frameworks that explicitly measure linguistic generalization to ensure reliable performance in real-world settings.