Don't Look at the Numbers: Visual Anchoring Bias and Layer-wise Representation in VLMs
For researchers and practitioners using VLMs, this work provides a causal account of visual anchoring bias and its link to representation dynamics, highlighting a systematic vulnerability in model judgments.
The paper demonstrates that embedded numeric anchors on images bias Vision-Language Model quality judgments across six VLMs, with anchor effects 2.5x larger than severe image quality degradation. Layer-wise probing shows a dissociation between anchor classification and quality prediction, revealing architecture-dependent integration patterns.
Embedded numeric anchors on images systematically bias Vision-Language Model quality judgments across six VLMs from five architectural families (ANOVA eta^2 = 0.18-0.77, all p < 0.001). Anchor effects are 2.5x larger than severe image quality degradation, confirming bias is not reducible to visual changes. Layer-wise probing reveals consistent dissociation: layers where anchor classification saturates (L12-L34) are suboptimal for quality prediction, with optimal layers deeper (R^2 = 0.69-0.91). Fusion analysis identifies architecture-dependent integration -- instant fusion at L1-L2 in two models versus partial or no fusion in three others. These results establish a causal account of visual anchoring bias, linking behavioral susceptibility to representation dynamics.