Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Models
This addresses hallucinations in LVLMs, which is a critical issue for improving model trustworthiness in applications like image captioning and visual QA, though it is an incremental improvement over existing training-free methods.
The paper tackles the problem of hallucinations in Large Vision-Language Models by proposing DCLA, a training-free decoding mechanism that aggregates layer representations to enforce consistency, resulting in reduced hallucinations and enhanced reliability on benchmarks like MME and POPE.
Despite the impressive capabilities of Large Vision-Language Models (LVLMs), they remain susceptible to hallucinations-generating content that is inconsistent with the input image. Existing training-free hallucination mitigation methods often suffer from unstable performance and high sensitivity to hyperparameter settings, limiting their practicality and broader adoption. In this paper, we propose a novel decoding mechanism, Decoding with Inter-layer Consistency via Layer Aggregation (DCLA), which requires no retraining, fine-tuning, or access to external knowledge bases. Specifically, our approach constructs a dynamic semantic reference by aggregating representations from previous layers, and corrects semantically deviated layers to enforce inter-layer consistency. The method allows DCLA to robustly mitigate hallucinations across multiple LVLMs. Experiments on hallucination benchmarks such as MME and POPE demonstrate that DCLA effectively reduces hallucinations while enhancing the reliability and performance of LVLMs.