ChartEditBench: Evaluating Grounded Multi-Turn Chart Editing in Multimodal Language Models
This addresses the need for better benchmarks to assess real-world interactive visualization capabilities in AI, though it is incremental as it builds on existing chart generation tasks.
The paper tackles the problem of evaluating multimodal language models' ability to handle iterative chart editing in exploratory data analysis, revealing that state-of-the-art models degrade substantially in multi-turn settings with frequent execution failures on data-centric transformations.
While Multimodal Large Language Models (MLLMs) perform strongly on single-turn chart generation, their ability to support real-world exploratory data analysis remains underexplored. In practice, users iteratively refine visualizations through multi-turn interactions that require maintaining common ground, tracking prior edits, and adapting to evolving preferences. We introduce ChartEditBench, a benchmark for incremental, visually grounded chart editing via code, comprising 5,000 difficulty-controlled modification chains and a rigorously human-verified subset. Unlike prior one-shot benchmarks, ChartEditBench evaluates sustained, context-aware editing. We further propose a robust evaluation framework that mitigates limitations of LLM-as-a-Judge metrics by integrating execution-based fidelity checks, pixel-level visual similarity, and logical code verification. Experiments with state-of-the-art MLLMs reveal substantial degradation in multi-turn settings due to error accumulation and breakdowns in shared context, with strong performance on stylistic edits but frequent execution failures on data-centric transformations. ChartEditBench, establishes a challenging testbed for grounded, intent-aware multimodal programming.