Visual Attention Drifts,but Anchors Hold:Mitigating Hallucination in Multimodal Large Language Models via Cross-Layer Visual Anchors
This addresses object hallucination in Multimodal Large Language Models, which is a critical reliability issue for applications like image captioning and visual QA, though it appears incremental as it builds on existing attention enhancement methods.
The paper tackles object hallucination in Multimodal Large Language Models by identifying that hallucination stems from deep layer attention regressing toward visual noise from early layers, and proposes CLVA, a training-free method that reinforces mid-layer features to pull attention back to correct visual regions, achieving outstanding performance across diverse architectures and benchmarks without significant computational overhead.
Multimodal Large Language Models often suffer from object hallucination. While existing research utilizes attention enhancement and visual retracing, we find these works lack sufficient interpretability regarding attention drift in final model stages. In this paper, we investigate the layer wise evolution of visual features and discover that hallucination stems from deep layer attention regressing toward initial visual noise from early layers. We observe that output reliability depends on acquiring visual anchors at intermediate layers rather than final layers. Based on these insights, we propose CLVA, which stands for Cross-Layer Visual Anchors, a training free method that reinforces critical mid layer features while suppressing regressive noise. This approach effectively pulls deep layer attention back to correct visual regions by utilizing essential anchors captured from attention dynamics. We evaluate our method across diverse architectures and benchmarks, demonstrating outstanding performance without significant increase in computational time and GPU memory.