Hacking Hallucinations of MLLMs with Causal Sufficiency and Necessity
This addresses hallucinations in MLLMs, which can generate semantically inconsistent outputs, improving reliability for vision-language applications, though it is incremental as it builds on existing causal analysis and optimization methods.
The paper tackled the problem of hallucinations in Multimodal Large Language Models (MLLMs) by proposing a reinforcement learning framework guided by causal completeness, which effectively mitigated hallucinations across various benchmark datasets and tasks.
Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities across vision-language tasks. However, they may suffer from hallucinations--generating outputs that are semantically inconsistent with the input image or text. Through causal analyses, we find that: (i) hallucinations with omission may arise from the failure to adequately capture essential causal factors, and (ii) hallucinations with fabrication are likely caused by the model being misled by non-causal cues. To address these challenges, we propose a novel reinforcement learning framework guided by causal completeness, which jointly considers both causal sufficiency and causal necessity of tokens. Specifically, we evaluate each token's standalone contribution and counterfactual indispensability to define a token-level causal completeness reward. This reward is used to construct a causally informed advantage function within the GRPO optimization framework, encouraging the model to focus on tokens that are both causally sufficient and necessary for accurate generation. Experimental results across various benchmark datasets and tasks demonstrate the effectiveness of our approach, which effectively mitigates hallucinations in MLLMs.