Mitigating Object Hallucination via Concentric Causal Attention
This addresses a critical reliability issue in multimodal AI systems where models generate inaccurate descriptions, though it appears incremental as it builds on known RoPE limitations.
The paper tackles object hallucination in Large Vision Language Models (LVLMs) by identifying Rotary Position Encoding (RoPE) long-term decay as a key cause and proposing Concentric Causal Attention (CCA), a positional alignment strategy that reduces relative distances between visual and instruction tokens, achieving significant improvements over existing methods on multiple benchmarks.
Recent Large Vision Language Models (LVLMs) present remarkable zero-shot conversational and reasoning capabilities given multimodal queries. Nevertheless, they suffer from object hallucination, a phenomenon where LVLMs are prone to generate textual responses not factually aligned with image inputs. Our pilot study reveals that object hallucination is closely tied with Rotary Position Encoding (RoPE), a widely adopted positional dependency modeling design in existing LVLMs. Due to the long-term decay in RoPE, LVLMs tend to hallucinate more when relevant visual cues are distant from instruction tokens in the multimodal input sequence. Additionally, we observe a similar effect when reversing the sequential order of visual tokens during multimodal alignment. Our tests indicate that long-term decay in RoPE poses challenges to LVLMs while capturing visual-instruction interactions across long distances. We propose Concentric Causal Attention (CCA), a simple yet effective positional alignment strategy that mitigates the impact of RoPE long-term decay in LVLMs by naturally reducing relative distance between visual and instruction tokens. With CCA, visual tokens can better interact with instruction tokens, thereby enhancing model's perception capability and alleviating object hallucination. Without bells and whistles, our positional alignment method surpasses existing hallucination mitigation strategies by large margins on multiple object hallucination benchmarks.