VACoDe: Visual Augmented Contrastive Decoding
This addresses the issue of hallucinations in LVLMs for users relying on accurate vision-language outputs, though it is incremental as it builds on existing contrastive decoding techniques.
The paper tackles the problem of inaccurate responses in Large Vision-Language Models by introducing VACoDe, a method that adaptively selects image augmentations for contrastive decoding, which outperforms previous methods and improves output quality across various vision-language tasks.
Despite the astonishing performance of recent Large Vision-Language Models (LVLMs), these models often generate inaccurate responses. To address this issue, previous studies have focused on mitigating hallucinations by employing contrastive decoding (CD) with augmented images, which amplifies the contrast with the original image. However, these methods have limitations, including reliance on a single augmentation, which is restrictive for certain tasks, as well as the high cost of using external knowledge. In this study, we address these limitations by exploring how to utilize multiple image augmentations. Through extensive experiments, we observed that different augmentations produce varying levels of contrast depending on the task. Based on this observation, we introduce a novel method called VACoDe, Visual Augmented Contrastive Decoding. This method adaptively selects the augmentation with the highest contrast for each task using the proposed softmax distance metric. Our empirical tests show that \alg outperforms previous methods and improves output quality in various vision-language tasks. Additionally, VACoDe can be universally applied across different model types and sizes without additional training or the use of external models and data.