Experience Augmented Policy Optimization for LLM Reasoning
For LLM reasoning, EAPO addresses the inefficiency and policy mismatch of experience reuse in RLVR, offering a more sample-efficient and effective training approach.
EAPO improves LLM reasoning by reusing prior RL-optimized policy as action-level experience, selectively injecting it at critical decision points, and using adapted importance sampling. It achieves consistent performance gains over SOTA RLVR methods on five benchmarks with Qwen models.
Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm for improving the reasoning capabilities of large language models (LLMs). However, existing RLVR methods typically rely on on-policy optimization from scratch, resulting in high sampling costs and inefficient utilization of accumulated experience. As model capabilities and policy behaviors evolve during training, recent attempts to reuse experience via fixed reasoning trajectories further suffer from policy mismatch. Motivated by these limitations, we argue that experience in RLVR should not be reused as fixed reasoning trajectories, but instead expressed in a policy-adaptive manner. In this work, we propose Experience-Augmented Policy Optimization (EAPO), which leverages a prior RL-optimized policy as an action-level experience prior and selectively injects experience at critical decision points during rollout. To ensure stable and unbiased learning from experience-augmented rollouts, EAPO further incorporates an adapted importance sampling scheme. Experiments on using Qwen-2.5-math 7b and Qwen-3-8B on five different benchmarks demonstrate that EAPO consistently improves reasoning performance over state-of-the-art RLVR methods.