EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understanding
This work addresses the challenge of automated chart comprehension for data analysis, providing a new benchmark and method, though it is incremental in improving existing visual language models.
The paper tackles the problem of limited high-quality training data and benchmarks for visual language models in chart understanding by introducing EvoChart, a self-training method for synthetic data generation, and EvoChart-QA, a benchmark with 650 real-world charts and 1,250 questions. The result shows that even the best proprietary model achieves only 49.8% accuracy on the benchmark, while the EvoChart method boosts open-source models to 54.2% accuracy.
Chart understanding enables automated data analysis for humans, which requires models to achieve highly accurate visual comprehension. While existing Visual Language Models (VLMs) have shown progress in chart understanding, the lack of high-quality training data and comprehensive evaluation benchmarks hinders VLM chart comprehension. In this paper, we introduce EvoChart, a novel self-training method for generating synthetic chart data to enhance VLMs' capabilities in real-world chart comprehension. We also propose EvoChart-QA, a noval benchmark for measuring models' chart comprehension abilities in real-world scenarios. Specifically, EvoChart is a unique self-training data synthesis approach that simultaneously produces high-quality training corpus and a high-performance chart understanding model. EvoChart-QA consists of 650 distinct real-world charts collected from 140 different websites and 1,250 expert-curated questions that focus on chart understanding. Experimental results on various open-source and proprietary VLMs tested on EvoChart-QA demonstrate that even the best proprietary model, GPT-4o, achieves only 49.8% accuracy. Moreover, the EvoChart method significantly boosts the performance of open-source VLMs on real-world chart understanding tasks, achieving 54.2% accuracy on EvoChart-QA.