Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formats
This addresses hallucination issues in LVLMs, which is a critical problem for improving reliability in vision-language tasks, though it appears incremental as it builds on existing transformer architectures.
The study tackled the problem of hallucinations in Large Vision-Language Models (LVLMs) by proposing an intervention framework that integrates effects across multiple causal pathways, resulting in consistent reduction of hallucinations across diverse alignment formats.
Despite their impressive performance across a wide range of tasks, Large Vision-Language Models (LVLMs) remain prone to hallucination. In this study, we propose a comprehensive intervention framework aligned with the transformer's causal architecture in LVLMs, integrating the effects of different intervention paths on hallucination. We find that hallucinations in LVLMs do not arise from a single causal path, but rather from the interplay among image-to-input-text, image-to-output-text, and text-to-text pathways. For the first time, we also find that LVLMs rely on different pathways depending on the question-answer alignment format. Building on these insights, we propose simple yet effective methods to identify and intervene on critical hallucination heads within each pathway, tailored to discriminative and generative formats. Experiments across multiple benchmarks demonstrate that our approach consistently reduces hallucinations across diverse alignment types.