Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models
For researchers evaluating vision-language models, this work provides a method to detect shortcut learning and assess true visual reasoning, revealing significant generalization failures.
The authors propose counterfactual charts to rigorously evaluate visual reasoning in VLMs, revealing that models often fail to generalize when charts are altered, especially when new visual reasoning is required.
Chart question-answering (QA) benchmarks aim to pose questions that require visual reasoning to correctly answer, but models can often reach solutions through shortcuts or prior familiarity with a chart based on their own background knowledge. To strictly evaluate visual reasoning, we propose counterfactual charts where the chart-question task remains fixed, but underlying chart and the corresponding answer are varied. We introduce Chartographer, a framework to reverse engineer charts into executable code, validate reconstruction fidelity, generate seed-controlled counterfactual variants, and derive new answers from executable QA logic. We apply this framework to existing chart QA datasets and evaluate proprietary and open-source vision-language models (VLMs), measuring variation sensitivity and generalizability. Counterfactual charts reveal failures hidden by single-chart performance: VLMs often fail to generalize after answering the original chart correctly. We find failures are most prevalent when updated charts require novel visual reasoning pathways.