Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration
This addresses a fundamental bottleneck in enhancing reasoning capabilities for LLMs, particularly in mathematical problem-solving, with incremental improvements over existing RLVR methods.
The paper tackled the limitation of reinforcement learning with verifiable rewards (RLVR) in large language models, where performance gains diminish with increased sampling due to restricted exploration from reverse KL divergence, and proposed RAPO to promote broader exploration, resulting in improved problem-solving on AIME2024 and AIME2025 datasets with Qwen2.5-3B and 7B models.
Reinforcement learning with verifiable rewards (RLVR) has recently enhanced the reasoning capabilities of large language models (LLMs), particularly for mathematical problem solving. However, a fundamental limitation remains: as the sampling budget increases, the advantage of RLVR-trained models over their pretrained bases often diminishes or even vanishes, revealing a strong dependence on the base model's restricted search space. We attribute this phenomenon to the widespread use of the reverse Kullback-Leibler (KL) divergence regularizer, whose mode-seeking behavior keeps the policy trapped inside the base model's support region and hampers wider exploration. To address this issue, we propose RAPO (Rewards-Aware Policy Optimization), an algorithm to promote broader yet focused exploration. Our method (i) utilizes the forward KL penalty to replace the reverse KL penalty for out-of-distribution exploration, and (ii) reweights the reference policy to facilitate adaptive in-distribution exploration. We train Qwen2.5-3B and 7B models with RAPO on the 8K SimpleRL-Zero dataset, without supervised fine-tuning, and evaluate them on AIME2024 and AIME2025. Results show that RAPO consistently improves problem-solving performance. Notably, RAPO enables models to surpass the base model's performance ceiling and solves previously intractable problems, advancing the frontier of RLVR for challenging reasoning tasks.