Mirage Probes: How Vision Models Fake Visual Understanding
For researchers and practitioners building trustworthy vision-language models, this work clarifies that mirage behavior is not a single failure mode but two, with different mitigation strategies, advancing understanding of visual grounding limitations.
The paper shows that vision-language models exhibit two distinct types of 'mirage' behavior—textual biases and spurious images—where they answer image-based questions without true visual grounding. Using a contrastive probing framework, the authors demonstrate that these behaviors are linearly decodable from internal activations and that spurious-image mirages require representational-level interventions, not just text cleaning.
Vision-language models (VLMs) can answer image-based questions confidently, and often correctly, even when no image is provided. This mirage behavior inflates benchmark scores without reflecting visual grounding. Prior work treats this as a single failure mode. We argue it is two. Using Mirage Probes, a contrastive probing framework that pairs paraphrased question variants with matched mirage and non-mirage labels on the same image, we show that mirage behavior is linearly decodable from internal activations across residual stream, MLP, post-attention, and attention-head sites in two open-source VLMs. We demonstrate that a Naive Bayes text baseline cannot recover this signal, ruling out surface lexical confounds. Cross-benchmark separability patterns, together with a novel Prior Harnessing Index (PHI) measuring how much a model can answer from text alone, expose two distinct regimes: textual biases, where the model answers from language priors without engaging visual representations, and spurious images, where it constructs false visual content in latent space and answers as if grounded. The distinction has direct mitigation consequences: text-distribution cleaning can address the first regime but cannot reach the second, since spurious-image mirages live in the model's visual representations rather than its text. Faithful visual grounding will require interventions at the representational level.