Spider 2.0-AIFunc: Extending Real-World Text-to-SQL to AI-Native SQL Workflows
For researchers and practitioners in text-to-SQL and AI-native database systems, this benchmark fills a gap by evaluating models on AI functions integrated into SQL, revealing that existing agent frameworks do not transfer effectively.
Spider 2.0-AIFunc extends text-to-SQL to AI-native SQL workflows, introducing a benchmark of 465 instances across 125 databases on Snowflake. The strongest proprietary models achieve 67-70% execution accuracy, while the best open-source model reaches 58.1%, with errors mainly in predicate specification and AI function parameterization.
Major cloud data platforms now expose large language model capabilities as native SQL functions, enabling analysts to perform classification, filtering, sentiment analysis, extraction, similarity search, and aggregation within ordinary SQL queries. Yet existing text-to-SQL benchmarks evaluate only conventional SQL and provide no signal on whether models can generate such AI-native SQL. We introduce Spider 2.0-AIFunc, a benchmark of 465 verified instances across 125 real-world databases covering six types of AI functions on the Snowflake platform. Starting from an existing enterprise text-to-SQL benchmark, we construct Spider 2.0-AIFunc through an agent-based pipeline that rewrites source tasks into AI-native form, simultaneously transforming target queries and refining natural language instructions to make the intended AI-native solution explicit and reduce ambiguity. All instances pass a multi-round repeated execution protocol across temporally separated windows to confirm result stability before release. Evaluating ten state-of-the-art language models, we find that the strongest proprietary models reach 67-70% execution accuracy while the best open-source model achieves 58.1%, a gap driven primarily by errors in predicate specification, schema grounding, and AI function parameterization. Agent frameworks designed for traditional text-to-SQL challenges, such as schema retrieval and relevant table selection, do not transfer effectively to AI-native SQL: a minimal agent setup consistently matches or outperforms more elaborate alternatives, suggesting that the strategies these frameworks employ are less critical in this setting. Data are available at https://github.com/Leolty/Spider2-AIFunc .