Both Ends Count! Just How Good are LLM Agents at "Text-to-Big SQL"?
This addresses a gap for practitioners using LLM agents in production-level big data workflows, though it is incremental as it focuses on evaluation metrics rather than a new method.
The paper tackles the problem of evaluating text-to-SQL systems in big data contexts, showing that existing metrics are insufficient and introducing new metrics that accurately reflect execution efficiency, cost, and data scale impact, with results including fine-grained comparisons of latency and cost across frontier models.
Text-to-SQL and Big Data are both extensively benchmarked fields, yet there is limited research that evaluates them jointly. In the real world, Text-to-SQL systems are often embedded with Big Data workflows, such as large-scale data processing or interactive data analytics. We refer to this as "Text-to-Big SQL". However, existing text-to-SQL benchmarks remain narrowly scoped and overlook the cost and performance implications that arise at scale. For instance, translation errors that are minor on small datasets lead to substantial cost and latency overheads as data scales, a relevant issue completely ignored by text-to-SQL metrics. In this paper, we overcome this overlooked challenge by introducing novel and representative metrics for evaluating Text-to-Big SQL. Our study focuses on production-level LLM agents, a database-agnostic system adaptable to diverse user needs. Via an extensive evaluation of frontier models, we show that text-to-SQL metrics are insufficient for Big Data. In contrast, our proposed text-to-Big SQL metrics accurately reflect execution efficiency, cost, and the impact of data scale. Furthermore, we provide LLM-specific insights, including fine-grained, cross-model comparisons of latency and cost.