DataClaw: A Process-Oriented Agent Benchmark for Exploratory Real-World Data Analysis
For researchers developing autonomous data analysis agents, DataClaw provides a more realistic and process-oriented evaluation benchmark that exposes current model weaknesses.
DataClaw is a benchmark for evaluating autonomous data analysis agents in exploratory, real-world settings with noisy data. It includes 2.06 million records and 492 tasks, finding that current LLM-based agents achieve below 50% accuracy, revealing significant limitations.
Evaluating autonomous data analysis agents requires testing their ability to perform exploratory analysis in underexplored data environments. However, many existing benchmarks emphasize final answer accuracy in prior-guided data settings and provide limited support for reasoning process evaluation. We introduce DataClaw, a process-oriented benchmark for exploratory real-world data analysis. DataClaw contains approximately 2.06 million real-world records across enterprise, industry and policy domains, with native data noise preserved. It further includes 492 cross-domain tasks derived from think-tank consulting scenarios, each annotated with intermediate milestones for process-level evaluation. These annotations allow DataClaw to measure how far an agent progresses and where its reasoning breaks down. Experiments with eight advanced LLMs show that current agents remain far from reliable in this setting, with seven models achieving below 50% overall accuracy. Process analysis further reveals partial progress hidden behind wrong answers and distinct exploration strategies across models. Overall, DataClaw provides a less data constrained diagnostic testbed for probing the capability boundaries of autonomous data-analysis agents.