CHARTOM: A Visual Theory-of-Mind Benchmark for LLMs on Misleading Charts
This addresses the need for better evaluation of LLMs in detecting misleading charts, which has societal benefits, but is incremental as it builds on existing benchmark methodologies.
The researchers tackled the problem of evaluating multimodal large language models' ability to understand and reason about misleading data visualizations by introducing CHARTOM, a visual theory-of-mind benchmark, and found it challenging for all tested models, including GPT, Claude, Gemini, Qwen, Llama, and Llava series, on both factual comprehension and misleadingness judgment tasks.
We introduce CHARTOM, a visual theory-of-mind benchmark designed to evaluate multimodal large language models' capability to understand and reason about misleading data visualizations though charts. CHARTOM consists of carefully designed charts and associated questions that require a language model to not only correctly comprehend the factual content in the chart (the FACT question) but also judge whether the chart will be misleading to a human readers (the MIND question), a dual capability with significant societal benefits. We detail the construction of our benchmark including its calibration on human performance and estimation of MIND ground truth called the Human Misleadingness Index. We evaluated several leading LLMs -- including GPT, Claude, Gemini, Qwen, Llama, and Llava series models -- on the CHARTOM dataset and found that it was challenging to all models both on FACT and MIND questions. This highlights the limitations of current LLMs and presents significant opportunity for future LLMs to improve on understanding misleading charts.