IDA-Bench: Evaluating LLMs on Interactive Guided Data Analysis
This addresses the need for better evaluation of LLMs in iterative data analysis for AI researchers, though it is incremental as it builds on existing benchmark concepts.
The paper tackles the problem of evaluating LLMs as data analysis agents by introducing IDA-Bench, a benchmark for multi-round interactive scenarios, and finds that state-of-the-art coding agents succeed on less than 50% of tasks.
Large Language Models (LLMs) show promise as data analysis agents, but existing benchmarks overlook the iterative nature of the field, where experts' decisions evolve with deeper insights of the dataset. To address this, we introduce IDA-Bench, a novel benchmark evaluating LLM agents in multi-round interactive scenarios. Derived from complex Kaggle notebooks, tasks are presented as sequential natural language instructions by an LLM-simulated user. Agent performance is judged by comparing its final numerical output to the human-derived baseline. Initial results show that even state-of-the-art coding agents (like Claude-3.7-thinking) succeed on < 50% of the tasks, highlighting limitations not evident in single-turn tests. This work underscores the need to improve LLMs' multi-round capabilities for building more reliable data analysis agents, highlighting the necessity of achieving a balance between instruction following and reasoning.