Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training Framework
For researchers and practitioners needing reliable chart data extraction, this work provides a benchmark and training method that significantly improves MLLM accuracy, though the approach is incremental.
The paper introduces a benchmark for evaluating multimodal LLMs on chart data extraction without visible labels, finding they struggle with precise value recovery. A training framework based on progressive learning achieves state-of-the-art performance with a 7B model, improving numerical accuracy and supporting mixed-initiative workflows.
Chart data extraction, which reverse-engineers data tables from chart images, is essential for reproducibility, analysis, retrieval, and redesign. Existing interactive tools are reliable but tedious, and mixed-initiative systems, while more efficient, lack generalizability. Recent multimodal large language models (MLLMs) offer a unified interface for chart interpretation, yet their ability to extract accurate data tables, especially without visible labels, remains unclear. We build a benchmark featuring diverse real-world charts without data labels to evaluate this capability. Results show that, while current MLLMs reliably reconstruct table structures, they struggle with precise value recovery. To address this, we revisit chart data extraction from a human-centered perspective and argue that extraction should follow a progressive learning process similar to how people read charts. Our training framework substantially improves numerical accuracy, achieving state-of-the-art performance with a 7B-parameter model. A user study further shows that our model effectively supports mixed-initiative workflows for reliable chart data extraction.