R$^2$PO: Decoupling Rollout and Inference Policies for LLM Reasoning
Improves reinforcement learning for LLM reasoning by addressing the entanglement of training and inference policies, benefiting researchers working on reasoning tasks.
R^2PO decouples training trajectories from inference responses in LLM reasoning, achieving 3.4% accuracy gain on MATH-500 and 1.3% on APPS with more diverse rollouts and reduced length bias.
Existing reinforcement learning methods for LLM reasoning implicitly assume that the policy generating training trajectories should coincide with the one producing inference responses. We argue that this is a misleading inductive bias: the optimization-optimal trajectory distribution favors informative gradients, whereas the inference-optimal response distribution emphasizes accuracy and consistency. Forcing both into a single policy entangles their gradients and suppresses exploration. We propose R$^2$PO (Residual Rollout Policy Optimization), which attaches a lightweight Residual Rollout-Head atop the policy to decouple training trajectories from inference responses, diversifying rollouts during training while keeping inference generation intact. Experiments show that R$^2$PO consistently outperforms baselines, with average accuracy gains of 3.4% on MATH-500 and 1.3% on APPS, alongside more diverse rollouts and reduced length bias. Our code is available at https://github.com/RRPO-ARR/Code.