Watermarking for Factuality: Guiding Vision-Language Models Toward Truth via Tri-layer Contrastive Decoding
This addresses the issue of hallucinations in large vision-language models, which is critical for improving reliability in multimodal AI applications, though it is an incremental improvement over existing methods.
The paper tackles the problem of hallucinations in large vision-language models by proposing a training-free tri-layer contrastive decoding method with watermarking, which achieves state-of-the-art performance on benchmarks like POPE, MME, and AMBER in reducing hallucinations and generating more visually grounded responses.
Large Vision-Language Models (LVLMs) have recently shown promising results on various multimodal tasks, even achieving human-comparable performance in certain cases. Nevertheless, LVLMs remain prone to hallucinations -- they often rely heavily on a single modality or memorize training data without properly grounding their outputs. To address this, we propose a training-free, tri-layer contrastive decoding with watermarking, which proceeds in three steps: (1) select a mature layer and an amateur layer among the decoding layers, (2) identify a pivot layer using a watermark-related question to assess whether the layer is visually well-grounded, and (3) apply tri-layer contrastive decoding to generate the final output. Experiments on public benchmarks such as POPE, MME and AMBER demonstrate that our method achieves state-of-the-art performance in reducing hallucinations in LVLMs and generates more visually grounded responses.