Do MLLMs Really Understand the Charts?
This addresses the issue of unreliable chart interpretation in MLLMs for users in data analysis and visualization, though it is incremental as it builds on existing MLLM frameworks.
The paper tackles the problem of Multimodal Large Language Models (MLLMs) hallucinating and performing poorly on non-annotated charts by proposing ChartReasoner, which mimics human visual reasoning; results show it outperforms models like GPT-4o and Gemini-2.5-Flash on a new benchmark and improves performance on public benchmarks.
Although Multimodal Large Language Models (MLLMs) have demonstrated increasingly impressive performance in chart understanding, most of them exhibit alarming hallucinations and significant performance degradation when handling non-annotated charts. Therefore, a question arises: Do MLLMs really understand the charts? Since a human is capable of understanding charts and estimating the values by visual reasoning, we first carefully establish a comprehensive Chart Reasoning Benchmark CRBench to rigorously evaluate the visual reasoning abilities of MLLMs on non-annotated charts. We argue that MLLMs are primarily relying on recognition rather than reasoning to interpret the charts. To steer MLLMs to reasonable chart understanding, we propose ChartReasoner that mimics human behavior by grounding their estimation in chart understanding. Extensive results on the proposed CRBench show that ChartReasnoner-3B/7B achieves superior performance in chart reasoning, even compared to GPT-4o and Gemini-2.5-Flash. More importantly, ChartReasnoner also demonstrates the visual reasoning abilities in general chart comprehension on public benchmarks, leading to significant performance gains and enabling MLLMs to rationally understand the charts. The code and dataset will be publicly available upon publication.