EchoRL: Reinforcement Learning via Rollout Echoing
This work is significant for researchers and practitioners using RLVR to improve large language models, as it tackles the problem of training signal collapse, which limits performance gains.
This paper addresses the problem of vanishing policy gradients in Reinforcement Learning with Verifiable Rewards (RLVR) for large language models, which occurs when all self-generated rollouts achieve verified success, leading to zero advantage. The authors propose EchoRL, a lightweight module that identifies an "EchoClip" from these advantage-degenerated rollouts based on step-level entropy and uses it as an auxiliary supervision signal, consistently improving RLVR post-training across 10 benchmarks, 5 LLM backbones, and 4 RLVR methods.
Reinforcement Learning with Verifiable Rewards is an effective route for post-training to strengthen the reasoning capability of large language models. However, as training proceeds, the learning signal can collapse thus makes the training gain become marginal and ineffective. Specifically, a growing fraction of prompts' rollouts become advantage-degenerated: all the self-generated rollouts show verified-success, making the standard deviation over their rewards be zero; accordingly each rollout's advantage becomes degenerated (zero) as well. Given such rollouts' advantages, the policy-gradient for model optimization eventually vanishes, capping the training performance. We argue that some of these rollouts still contain valuable learning signals but unfortunately omitted with the existing RLVR methods. In this paper, inspired through analyzing the entropy pattern behind golden trajectories produced by external expert models, we propose EchoRL for better exploiting the advantage-degenerated rollouts to further improve the training performance. EchoRL is a lightweight module that first identifies an EchoClip from verified-success rollouts based on their step-level entropy values, and then feeds this clip back as an auxiliary supervision signal in the RL objective. Extensive experiments across 10 benchmarks, 5 LLM backbones, and 4 popular RLVR post-training methods demonstrate that EchoRL consistently improves RLVR post-training with minimal overhead.