Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities
For researchers and practitioners evaluating LLMs for data analysis, this benchmark highlights the inadequacy of current models on realistic, multi-tabular, and insight-driven tasks.
The authors introduce DataGovBench, a benchmark derived from governmental open data to evaluate LLMs on real-world data analysis tasks including complex Table QA and exploratory Table Insight. Experiments show significant performance gaps, indicating current LLMs are far from meeting real-world data analytics demands.
Current benchmarks for evaluating Large Language Models (LLMs) in data analysis often fail to reflect real-world settings. They typically focus on fact retrieval from small tables and overlook the challenges of large multi-tabular datasets, external knowledge integration, and exploratory insight discovery. We introduce DataGovBench, a benchmark derived from governmental open data designed to evaluate LLMs in practical scenarios. The benchmark includes two tasks: Table QA that requires solving complex decomposable questions and producing textual answers or visualizations, and Table Insight that evaluates the ability of models to generate expert-level findings through exploratory data analysis. Comprehensive experiments with state-of-the-art LLMs, both with and without agentic frameworks, reveal significant performance gaps across both tasks. These results suggest that current LLM-based systems remain far from satisfying the demands of real-world data analytics. DataGovBench provides a challenging benchmark for advancing research on LLMs capable of both answering analytical queries and discovering insights from data. Code and sample data are available at https://github.com/SoHasegawa/datagovbench.