Be Faithful When Response: Returning Fluent and Grounded Answers for Vision-Language Models Reinforcement Learning
For researchers applying RL to vision-language models, this work provides a method to mitigate visually ungrounded reasoning, though it is an incremental improvement over existing RL fine-tuning approaches.
The paper addresses instability in reinforcement learning for vision-language models due to exploitation of language priors and neglect of visual evidence. The proposed Faithful Warm-Start (FWS) strategy curates a dataset with explicit vision-language causal relationships and uses a VLM-based judge to ensure visual faithfulness, improving answer accuracy and stabilizing RL training.
Reinforcement Learning (RL) is an important paradigm for improving the reasoning capabilities of Vision-Language Models (VLMs). However, directly applying RL to rollout multimodal reasoning can lead to instability, due to the exploitation of language priors, the neglect of visual evidence, and the generation of reasoning traces that are fluent yet not visually grounded. The question arises: Can initially steer the policy toward visually faithful reasoning regime before applying reinforcement learning? To this end, we propose a Faithful Warm-Start (FWS) strategy that first curates samples with explicit vision-language causal relationships from six general VQA benchmarks to construct the FaithfulQA dataset, where each of the image-question pairs gains a certain degree of visual observations, question requirements, commonsense knowledge, domain knowledge, and the final answer. Subsequently, a VLM-based judge is employed to further purify the dataset, ensuring strong causal consistency and visual faithfulness. This warm-start stage equips the model with the capability to understand causally grounded vision-language patterns before subsequent RL optimization under sparse answer-level rewards. Experimental results show that such faithful supervision improves answer accuracy, stabilizes RL training, and reduces visually unsupported reasoning.