V-ITI: Mitigating Hallucinations in Multimodal Large Language Models via Visual Inference-Time Intervention
This addresses reliability issues in precision-sensitive domains like healthcare or autonomous systems by mitigating hallucinations, though it is an incremental improvement over existing intervention methods.
The paper tackles the problem of hallucinations in Multimodal Large Language Models (MLLMs) by proposing V-ITI, a lightweight framework that detects visual neglect and intervenes only when necessary, reducing hallucinations across eight benchmarks while maintaining general performance.
Multimodal Large Language Models (MLLMs) excel in numerous vision-language tasks yet suffer from hallucinations, producing content inconsistent with input visuals, that undermine reliability in precision-sensitive domains. This issue stems from a fundamental problem of visual neglect, where models fail to adequately prioritize input images. Existing methods typically alleviate hallucinations by intervening in the attention score or output logits, focusing on "how to intervene" but overlooking the prerequisite "when to intervene", which leads to the "over-intervention" problem and subsequently introduces new hallucinations and unnecessary computational overhead. To address this gap, we first investigate the mechanism of visual neglect and reveal it can be accurately detected via head-level activation patterns in MLLMs. We thus propose V-ITI, a lightweight visual inference-time intervention framework integrating a Visual Neglect Detector that identifies visual neglect via head-level discriminative probes and a Visual Recall Intervenor that modulates activations with prestored visual activation information only when the visual neglect is detected. Extensive experiments across eight benchmarks and different MLLM families demonstrate that V-ITI consistently mitigates vision-related hallucinations while preserving general task performance.