StatABench: Dataset and Framework for Evaluating Statistical Analysis Capabilities of LLMs
For researchers and practitioners assessing LLMs in data science, this benchmark provides a more comprehensive and challenging evaluation than prior limited-scope benchmarks.
StatABench is a benchmark for evaluating LLMs' statistical analysis capabilities, comprising 404 closed-form questions and 30 open-ended modeling tasks. Experiments show GPT-5.1 achieves only 68.6% on closed tasks and the best agent scores 61.86 on open tasks, revealing significant gaps in tool-grounded reasoning and methodological decision-making.
Statistical analysis is a broad, complex field requiring both domain knowledge and tool proficiency. While prior work has evaluated large language models (LLMs) in this domain, existing benchmarks remain limited in scope and format. To bridge this gap, we introduce StatABench (Statistical AnalysisBenchmark), a benchmark designed to systematically assess LLMs' statistical analysis capabilities. StatABench comprises two complementary components: Stat-Closed, containing 404 questions across 18 statistical topics in multiple formats (multiple-choice, fill-in-the-blank, decision-making, and practical application), and Stat-Open, featuring 30 complex open-ended modeling tasks adapted from professional competitions. We evaluate diverse LLMs using the LangChain MCP framework and multiple data science agents, and assess Stat-Open solutions via a validated LLM-as-Judge protocol. Experiments show that even GPT-5.1 achieves only 68.6% on Stat-Closed, while the best open-source model reaches 60.6%. On Stat-Open, the top agent framework scores 61.86 on average. These results reveal the gap between current LLMs and reliable statistical analysis, highlighting persistent challenges in tool-grounded reasoning, methodological decision-making, and end-to-end statistical modeling.