Delineating Knowledge Boundaries for Honest Large Vision-Language Models
For developers of trustworthy VLMs, this work provides a method to reduce hallucinations by teaching models to recognize and refuse unknown questions, though the gains are incremental.
This paper addresses factual hallucinations in large vision-language models (VLMs) by enhancing their ability to refuse queries beyond their knowledge. Using a model-specific dataset and preference-aware optimization, they improved the Truthful Rate from 57.9% to 67.3% on the Visual-Idk dataset.
Large Vision-Language Models (VLMs) have achieved remarkable multimodal performance yet remain prone to factual hallucinations, particularly in long-tail or specialized domains. Moreover, current models exhibit a weak capacity to refuse queries that exceed their parametric knowledge. In this paper, we propose a systematic framework to enhance the refusal capability of VLMs when facing such unknown questions. We first curate a model-specific "Visual-Idk" (Visual-I don't know) dataset, leveraging multi-sample consistency probing to distinguish between known and unknown facts. We then align the model using supervised fine-tuning followed by preference-aware optimization (e.g., DPO, ORPO) to effectively delineate its knowledge boundaries. Results on the Visual-Idk dataset show our method improves the Truthful Rate from 57.9\% to 67.3\%. Additionally, internal probing also demonstrates that the model genuinely recognizes its boundaries instead of just memorizing refusal patterns. Our framework further generalizes to out-of-distribution medical and perceptual domains, providing a robust path toward more trustworthy and prudent visual assistants.