When Visuals Aren't the Problem: Evaluating Vision-Language Models on Misleading Data Visualizations
This work addresses the challenge of misinformation propagation through deceptive visualizations for users relying on VLMs, but it is incremental as it builds on existing chart understanding tasks by introducing a fine-grained benchmark.
The paper tackled the problem of Vision-Language Models (VLMs) struggling to detect misleading data visualizations, especially when deception arises from subtle reasoning errors in captions, and found that models detect visual design errors more reliably than reasoning-based misinformation, with frequent misclassifications of non-misleading visualizations.
Visualizations help communicate data insights, but deceptive data representations can distort their interpretation and propagate misinformation. While recent Vision Language Models (VLMs) perform well on many chart understanding tasks, their ability to detect misleading visualizations, especially when deception arises from subtle reasoning errors in captions, remains poorly understood. Here, we evaluate VLMs on misleading visualization-caption pairs grounded in a fine-grained taxonomy of reasoning errors (e.g., Cherry-picking, Causal inference) and visualization design errors (e.g., Truncated axis, Dual axis, inappropriate encodings). To this end, we develop a benchmark that combines real-world visualization with human-authored, curated misleading captions designed to elicit specific reasoning and visualization error types, enabling controlled analysis across error categories and modalities of misleadingness. Evaluating many commercial and open-source VLMs, we find that models detect visual design errors substantially more reliably than reasoning-based misinformation, and frequently misclassify non-misleading visualizations as deceptive. Overall, our work fills a gap between coarse detection of misleading content and the attribution of the specific reasoning or visualization errors that give rise to it.