When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models
For developers and users of vision-language models, this work provides a mechanistic understanding and practical, low-cost solutions to mitigate hallucinations in high-stakes applications.
The paper identifies geometric over-alignment as the root cause of hallucinations in decoder-based VLMs, where visual embeddings are overly aligned with a linguistic bias subspace. The proposed training-free and fine-tuning debiasing methods reduce hallucinations across multiple benchmarks (e.g., POPE, CHAIR, AMBER) and improve CLAIR scores without added computational cost.
Vision-Language Models (VLMs) increasingly power high-stakes applications, from medical imaging to autonomous systems, yet they routinely hallucinate, confidently describing content not present in the input. We investigate the root causes of these failure modes with a mechanistic analysis focusing on the decoder-based VLMs. We trace these failure modes to a geometric over-alignment: to bridge the modality gap required by attention mechanisms, decoder-based VLMs over-align visual embeddings with the text manifold, injecting a statistical linguistic bias that systematically overshadows fine-grained visual evidence. While prior work either aggressively closes this gap or suppresses hallucinations through expensive black-box decoding strategies, none addresses the underlying geometric cause. We provide the first quantitative characterization of this over-alignment, demonstrating that linguistic bias concentrates in the top principal components of a universal, dataset-agnostic text subspace. Building on this insight, we propose two complementary remedies: a training-free inference strategy and a bias-aware fine-tuning paradigm, both of which explicitly project out this subspace from visual representations. Our methods significantly reduce hallucinations across POPE, CHAIR, and AMBER benchmarks, and improve CLAIR scores on long-form captioning tasks, with the training-free variant adding no computational overhead over the base model.