VL-Calibration: Decoupled Confidence Calibration for Large Vision-Language Models Reasoning
This addresses reliability issues for users in high-stakes domains by reducing hallucinations and improving confidence estimation in multimodal AI systems, though it is an incremental advancement over existing calibration methods.
The paper tackles the problem of large vision-language models (LVLMs) producing incorrect responses with high certainty by proposing VL-Calibration, a reinforcement learning framework that decouples confidence into visual and reasoning components, which improves calibration and boosts visual reasoning accuracy across thirteen benchmarks.
Large Vision Language Models (LVLMs) achieve strong multimodal reasoning but frequently exhibit hallucinations and incorrect responses with high certainty, which hinders their usage in high-stakes domains. Existing verbalized confidence calibration methods, largely developed for text-only LLMs, typically optimize a single holistic confidence score using binary answer-level correctness. This design is mismatched to LVLMs: an incorrect prediction may arise from perceptual failures or from reasoning errors given correct perception, and a single confidence conflates these sources while visual uncertainty is often dominated by language priors. To address these issues, we propose VL-Calibration, a reinforcement learning framework that explicitly decouples confidence into visual and reasoning confidence. To supervise visual confidence without ground-truth perception labels, we introduce an intrinsic visual certainty estimation that combines (i) visual grounding measured by KL-divergence under image perturbations and (ii) internal certainty measured by token entropy. We further propose token-level advantage reweighting to focus optimization on tokens based on visual certainty, suppressing ungrounded hallucinations while preserving valid perception. Experiments on thirteen benchmarks show that VL-Calibration effectively improves calibration while boosting visual reasoning accuracy, and it generalizes to out-of-distribution benchmarks across model scales and architectures.