ConDABench: Interactive Evaluation of Language Models for Data Analysis
This provides a new benchmark for evaluating conversational data analysis tools, addressing a gap in existing evaluations, but it is incremental as it builds on prior work in benchmarking.
The authors tackled the problem of evaluating language models for interactive data analysis by introducing ConDABench, a framework that generates conversational benchmarks from public datasets, resulting in 1,420 problems and showing that newer models solve more instances but not necessarily long-form tasks.
Real-world data analysis tasks often come with under-specified goals and unclean data. User interaction is necessary to understand and disambiguate a user's intent, and hence, essential to solving these complex tasks. Existing benchmarks for evaluating LLMs on data analysis tasks do not capture these complexities or provide first-class support for interactivity. We introduce ConDABench, a framework for generating conversational data analysis (ConDA) benchmarks and evaluating external tools on the generated benchmarks. \bench consists of (a) a multi-agent workflow for generating realistic benchmarks from articles describing insights gained from public datasets, (b) 1,420 ConDA problems generated using this workflow, and (c) an evaluation harness that, for the first time, makes it possible to systematically evaluate conversational data analysis tools on the generated ConDA problems. Evaluation of state-of-the-art LLMs on the benchmarks reveals that while the new generation of models are better at solving more instances, they are not necessarily better at solving tasks that require sustained, long-form engagement. ConDABench is an avenue for model builders to measure progress towards truly collaborative models that can complete complex interactive tasks.