Two Causes, Not One: Rethinking Omission and Fabrication Hallucinations in MLLMs
This addresses a critical oversight in current research for improving robustness in MLLMs, though it is incremental as it builds on existing hallucination mitigation efforts.
The paper tackles the problem of object hallucination in Multimodal Large Language Models by showing that omission and fabrication hallucinations have distinct causes, and it introduces Visual Potential Field Calibration to reduce omissions without increasing fabrications.
Multimodal Large Language Models (MLLMs) have achieved impressive advances, yet object hallucination remains a persistent challenge. Existing methods, based on the flawed assumption that omission and fabrication hallucinations share a common cause, often reduce omissions only to trigger more fabrications. In this work, we overturn this view by demonstrating that omission hallucinations arise from insufficient confidence when mapping perceived visual features to linguistic expressions, whereas fabrication hallucinations result from spurious associations within the cross-modal representation space due to statistical biases in the training corpus. Building on findings from visual attention intervention experiments, we propose the Visual-Semantic Attention Potential Field, a conceptual framework that reveals how the model constructs visual evidence to infer the presence or absence of objects. Leveraging this insight, we introduce Visual Potential Field Calibration (VPFC), a plug-and-play hallucination mitigation method that effectively reduces omission hallucinations without introducing additional fabrication hallucinations. Our findings reveal a critical oversight in current object hallucination research and chart new directions for developing more robust and balanced hallucination mitigation strategies.