Efficient Contrastive Decoding with Probabilistic Hallucination Detection - Mitigating Hallucinations in Large Vision Language Models -
This addresses hallucinations in LVLMs, which is a critical issue for reliable AI applications, though it is an incremental improvement as it builds on existing decoding methods without requiring model retraining.
The paper tackles the problem of hallucinatory responses in Large Vision Language Models (LVLMs) by introducing Efficient Contrastive Decoding (ECD), which uses probabilistic hallucination detection to suppress hallucinations at inference time, resulting in outperforming state-of-the-art methods on benchmarks with improved computation time.
Despite recent advances in Large Vision Language Models (LVLMs), these models still suffer from generating hallucinatory responses that do not align with the visual input provided. To mitigate such hallucinations, we introduce Efficient Contrastive Decoding (ECD), a simple method that leverages probabilistic hallucination detection to shift the output distribution towards contextually accurate answers at inference time. By contrasting token probabilities and hallucination scores, ECD subtracts hallucinated concepts from the original distribution, effectively suppressing hallucinations. Notably, our proposed method can be applied to any open-source LVLM and does not require additional LVLM training. We evaluate our method on several benchmark datasets and across different LVLMs. Our experiments show that ECD effectively mitigates hallucinations, outperforming state-of-the-art methods with respect to performance on LVLM benchmarks and computation time.