VisualFLIP: Do Predictions Depend on Task-Critical Visual Evidence in Multimodal Reasoning?
For researchers evaluating multimodal reasoning, this benchmark reveals that accuracy alone is insufficient to assess grounding, highlighting a critical failure mode in current MLLMs.
The paper introduces VisualFLIP, a paired benchmark of 1,374 images with same-question perturbation pairs to test whether multimodal LLMs rely on task-critical visual evidence. Evaluation of 24 models shows that correct answers often coexist with flawed reasoning, and many models fail to update predictions after critical visual changes, with collapse rates increasing in sequential settings.
When a multimodal large language model answers a visual reasoning question correctly, is the prediction actually supported by the task-critical visual evidence? Correct answers can coexist with flawed reasoning, making accuracy alone an incomplete test of grounding. We introduce VisualFLIP, a paired benchmark with 1,374 images arranged as same-question perturbation pairs across cardinality, attribute, spatial, and logic tasks. Each pair keeps the question fixed but minimally changes the evidence so the gold answer deterministically flips. We evaluate 24 MLLMs with pair accuracy, which requires solving both sides of a pair, and Collapse Rate (CR), which measures how often a model that solves at least one side repeats the same non-empty answer for both images. Together, these metrics show that paired correctness and evidence dependence are related but distinct: capable models can still fail to update after task-critical visual changes, and collapse becomes more severe for some models when the edited image follows an earlier answer in a sequential setting. Further details are available on our project page: https://didizhu-judy.github.io/VisualFLIP/