Improving Generalization Robustness of Multimodal RLVR
This work provides a more robust post-training method for Multimodal Large Language Models, particularly important for reliable deployment in high-stakes applications such as medical VQA, by improving generalization robustness to prompt variations.
The paper addresses the brittleness of Reinforcement Learning with Verifiable Rewards (RLVR) in Multimodal Large Language Models, where performance degrades with paraphrased questions or prompt changes. They propose Prompt-Invariant RLVR (PIRL), which uses a dynamic trinary reward and a consistency regularizer, resulting in an average accuracy drop of only ≤1% under stress testing, compared to ~3% for GRPO.
Reinforcement Learning with Verifiable Rewards (RLVR) makes Multimodal Large Language Models more accurate, but the gains are brittle: simply paraphrasing a question or changing the prompt template can degrade them, which challenges reliable deployment in high-stakes scenarios like medical VQA. We trace this to two issues of the standard RL objective. First, the binary verifier conflates format with content, so the reward signal cannot tell a wrong answer apart from a misformatted one. Second, the training distribution covers only a thin slice of the real-world prompts that the model might meet at deployment, so policies that perform well on the training distribution can behave differently under unseen prompts during test. Both failures call for a robust post-training method that helps the policy cover a broader distribution of semantically equivalent prompts, and we identify two measures that help achieve this objective: separating format from semantics in the reward, and applying policy invariance across perturbed prompts with equivalent semantics. We therefore propose Prompt-Invariant RLVR (PIRL), consisting of a dynamic trinary reward and a consistency regularizer based on an embedding-space adversary. Under stress testing, PIRL's average accuracy on benchmarks drops by only $\le 1\%$, where GRPO drops ~3%. On dynamic evaluation, PIRL also achieves the smallest performance drop.