RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Training
This work addresses a key bottleneck in enhancing reasoning capabilities for large language models, offering a novel optimization paradigm that could benefit AI applications requiring robust and diverse outputs.
The paper tackled the problem of entropy collapse and limited reasoning gains in reinforcement learning for post-training large language models by proposing Risk-based Policy Optimization (RiskPO), which achieved consistent and significant improvements in mathematical reasoning, multi-modal reasoning, and code generation benchmarks, surpassing existing methods on Pass@1 and Pass@k metrics.
Reinforcement learning with verifiable reward has recently emerged as a central paradigm for post-training large language models (LLMs); however, prevailing mean-based methods, such as Group Relative Policy Optimization (GRPO), suffer from entropy collapse and limited reasoning gains. We argue that these issues stem from overemphasizing high-probability output sequences while neglecting rare but informative reasoning paths. To address these challenges, we propose Risk-based Policy Optimization (RiskPO), which substitutes classical mean-based objectives with principled risk measures. Specifically, we introduce a Mixed Value-at-Risk objective that integrates weighted attention over multiple regions of the reward distribution, thereby amplifying gradient signals on challenging instances and preventing overconfident convergence. We further design a bundling scheme that aggregates multiple questions into bundles, thus enriching the feedback signal and yielding more stable and informative training dynamics. Theoretically, we prove that the risk-averse update alleviates entropy collapse and promotes exploration. Numerically, RiskPO achieves consistent and significant improvements in mathematical reasoning, multi-modal reasoning, and code generation benchmarks, surpassing GRPO and its variants on both Pass@1 and Pass@k metrics. Our results demonstrate that risk-based optimization provides a rigorous and effective paradigm for enhancing LLM reasoning capabilities.