Uncalibrated Reasoning: GRPO Induces Overconfidence for Stochastic Outcomes
This addresses a critical issue for researchers applying RL to language models in stochastic reasoning tasks, though it is incremental as it builds on existing methods.
The paper tackled the problem of reinforcement learning methods inducing overconfident predictions in language models for stochastic domains like scientific experiments, showing that GRPO causes miscalibration while PPO and RLOO yield well-calibrated models, with removal of group standard normalization fixing the issue.
Reinforcement learning (RL) has proven remarkably effective at improving the accuracy of language models in verifiable and deterministic domains like mathematics. Here, we examine if current RL methods are also effective at optimizing language models in verifiable domains with stochastic outcomes, like scientific experiments. Through applications to synthetic data and real-world biological experiments, we demonstrate that Group Relative Policy Optimization (GRPO) induces overconfident probability predictions for binary stochastic outcomes, while Proximal Policy Optimization (PPO) and REINFORCE Leave-One-Out (RLOO) yield well-calibrated models. We show that removing group standard normalization in GRPO fixes its miscalibration and provide a theoretical explanation for why normalization causes overconfidence. Our results provide new evidence against the use of standard normalization in GRPO and help pave the way for applications of RL for reasoning language models beyond deterministic domains.