VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative Training
This work addresses alignment issues in vision-language models for applications requiring reliable multimodal verification, though it is incremental in enhancing existing methods.
The paper tackles the challenges of training Vision-Language Reward Models (VL-RMs), such as the bootstrapping dilemma and modality bias, by proposing an iterative framework using vision experts, Chain-of-Thought rationales, and Margin-based Rejection Sampling. The result is improved performance in hallucination detection and multimodal reasoning on VL-RM benchmarks.
Reinforcement Fine-Tuning (RFT) with verifiable rewards has advanced large language models but remains underexplored for Vision-Language (VL) models. The Vision-Language Reward Model (VL-RM) is key to aligning VL models by providing structured feedback, yet training effective VL-RMs faces two major challenges. First, the bootstrapping dilemma arises as high-quality training data depends on already strong VL models, creating a cycle where self-generated supervision reinforces existing biases. Second, modality bias and negative example amplification occur when VL models hallucinate incorrect visual attributes, leading to flawed preference data that further misguides training. To address these issues, we propose an iterative training framework leveraging vision experts, Chain-of-Thought (CoT) rationales, and Margin-based Rejection Sampling. Our approach refines preference datasets, enhances structured critiques, and iteratively improves reasoning. Experiments across VL-RM benchmarks demonstrate superior performance in hallucination detection and multimodal reasoning, advancing VL model alignment with reinforcement learning.