Token Preference Optimization with Self-Calibrated Visual-Anchored Rewards for Hallucination Mitigation
This addresses hallucinations in vision-language models, which is a critical issue for reliable AI applications, though it appears incremental as it builds on existing Direct Preference Optimization methods.
The paper tackles the problem of mitigating hallucinations in Large Vision Language Models by proposing Token Preference Optimization with self-calibrated visual-anchored rewards, which achieves state-of-the-art performance with absolute improvements on hallucination benchmarks.
Direct Preference Optimization (DPO) has been demonstrated to be highly effective in mitigating hallucinations in Large Vision Language Models (LVLMs) by aligning their outputs more closely with human preferences. Despite the recent progress, existing methods suffer from two drawbacks: 1) Lack of scalable token-level rewards; and 2) Neglect of visual-anchored tokens. To this end, we propose a novel Token Preference Optimization model with self-calibrated rewards (dubbed as TPO), which adaptively attends to visual-correlated tokens without fine-grained annotations. Specifically, we introduce a token-level \emph{visual-anchored} \emph{reward} as the difference of the logistic distributions of generated tokens conditioned on the raw image and the corrupted one. In addition, to highlight the informative visual-anchored tokens, a visual-aware training objective is proposed to enhance more accurate token-level optimization. Extensive experimental results have manifested the state-of-the-art performance of the proposed TPO. For example, by building on top of LLAVA-1.5-7B, our TPO boosts the performance absolute improvement for hallucination benchmarks.