FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Models
For developers of large vision-language models, this work provides a mechanistic understanding and a simple, training-free solution to mitigate hallucinations, though the improvement is incremental over existing contrastive decoding methods.
The paper identifies FFN modules in critical layers of LVLMs as the source of language priors that cause hallucinations, and proposes FADE, a training-free method that attenuates FFN outputs to reduce this dominance. FADE reduces hallucinations on POPE, CHAIR, and MME benchmarks across multiple models while preserving inference efficiency.
Despite the impressive capabilities of Large Vision-Language Models (LVLMs), they remain susceptible to hallucination, generating content inconsistent with the input image. Recent studies attribute this to the dominance of language priors over visual inputs and employ contrastive decoding methods to mitigate this dominance, but the mechanistic origin remains unexplored. We investigate the information flow through each transformer layer and find that attention modules consistently aggregate visual evidence, while FFN modules at critical layers act as the source of language priors. These priors can override visual evidence, causing correct predictions in intermediate layers to drift toward incorrect outputs. Based on this insight, we propose FADE (FFN Attenuation for DEcoding), a training-free method that attenuates FFN outputs to reduce language-prior dominance. Evaluations on POPE, CHAIR, and MME benchmarks across LLaVA-1.5, mPLUG-Owl2, and InstructBLIP show that FADE effectively mitigates hallucinations while preserving inference efficiency.