Watch Closely: Mitigating Object Hallucinations in Large Vision-Language Models with Disentangled Decoding
This addresses a critical issue for real-world applications of LVLMs, such as in safety-sensitive domains, by mitigating hallucinations that can lead to erroneous outputs, though it is an incremental improvement over existing methods that focus only on language modality.
The paper tackles the problem of object hallucinations in Large Vision-Language Models (LVLMs), where models generate fluent but inaccurate text due to misidentifying objects, and introduces the Hallucination Disentangled Decoding (HDD) method, which reduces hallucinations in both language and visual modalities without requiring training.
Large Vision-Language Models (LVLMs) bridge the gap between visual and linguistic modalities, demonstrating strong potential across a variety of domains. However, despite significant progress, LVLMs still suffer from severe hallucination issues in object recognition tasks. These models often fail to accurately identify certain objects, leading to text generation that appears fluent but does not correspond to the visual content, which can have serious consequences in real-world applications. Recently, several methods have been proposed to alleviate LVLM hallucinations, but most focus solely on reducing hallucinations in the language modality. To mitigate hallucinations in both the language and visual modalities, we introduce Hallucination Disentangled Decoding (HDD) method that requires no training. HDD enhances the original image by segmenting it and selecting images that augment the original, while also utilizing a blank image to eliminate language prior hallucinations in both the original and segmented images. This design not only reduces the model's dependence on language priors but also enhances its visual performance. (Code: https://github.com/rickeyhhh/Hallucination-Disentangled-Decoding)