ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLM
This addresses the issue of unreliable and unsafe outputs in MLLMs for users in applications like visual question answering, offering a model-agnostic solution with incremental improvements over prior methods.
The paper tackles the problem of hallucination in multimodal large language models (MLLMs) by proposing Attention-Steerable Contrastive Decoding (ASCD), which reduces hallucination by up to 38.2% on benchmarks like POPE, CHAIR, and MMHal-Bench while improving accuracy on standard VQA tasks.
Multimodal large language models (MLLMs) frequently hallucinate by over-committing to spurious visual cues. Prior remedies-Visual and Instruction Contrastive Decoding (VCD, ICD)-mitigate this issue, yet the mechanism remains opaque. We first empirically show that their improvements systematically coincide with redistributions of cross-modal attention. Building on this insight, we propose Attention-Steerable Contrastive Decoding (ASCD), which directly steers the attention scores during decoding. ASCD combines (i) positive steering, which amplifies automatically mined text-centric heads-stable within a model and robust across domains-with (ii) negative steering, which dampens on-the-fly identified critical visual tokens. The method incurs negligible runtime and memory overhead and requires no additional training. Across five MLLM backbones and three decoding schemes, ASCD reduces hallucination on POPE, CHAIR, and MMHal-Bench by up to 38.2 percent while improving accuracy on standard VQA benchmarks, including MMMU, MM-VET, ScienceQA, TextVQA, and GQA. These results position attention steering as a simple, model-agnostic, and principled route to safer, more faithful multimodal generation.