VCE: A zero-cost hallucination mitigation method of LVLMs via visual contrastive editing
For practitioners deploying LVLMs in accuracy-critical domains like medical imaging and autonomous driving, VCE offers a scalable, label-free method to reduce object hallucination while maintaining computational efficiency.
VCE reduces object hallucination in large vision-language models by identifying and suppressing hallucinatory tendencies via contrastive visual perturbations and SVD-based parameter editing, achieving effective mitigation without fine-tuning or labeled data.
Large vision-language models (LVLMs) frequently suffer from Object Hallucination (OH), wherein they generate descriptions containing objects that are not actually present in the input image. This phenomenon is particularly problematic in real-world applications such as medical imaging and autonomous driving, where accuracy is critical. Recent studies suggest that the hallucination problem may stem from language priors: biases learned during pretraining that cause LVLMs to generate words based on their statistical co-occurrence. To mitigate this problem, we propose Visual Contrastive Editing (VCE), a novel post-hoc method that identifies and suppresses hallucinatory tendencies by analyzing the model's response to contrastive visual perturbations. Using Singular Value Decomposition (SVD), we decompose the model's activation patterns to isolate hallucination subspaces and apply targeted parameter edits to attenuate its influence. Unlike existing approaches that require fine-tuning or labeled data, VCE operates as a label-free intervention, making it both scalable and practical for deployment in resource-constrained settings. Experimental results demonstrate that VCE effectively reduces object hallucination across multiple benchmarks while maintaining the model's original computational efficiency.