Rethinking Groups in Critic-Free RLVR
For practitioners training LLMs with RL, this work reduces data and synchronization overhead while maintaining or improving performance.
The paper identifies that group-based advantage estimation in critic-free RL for LLMs is inefficient and inflexible, and proposes negative token filtering to enable stable single-rollout training, achieving comparable or better performance on reasoning and agentic tasks.
Reinforcement learning (RL) has become a central paradigm for post-training large language models. Existing critic-free RL methods typically generate a group of rollouts for the same question to estimate value baselines for advantage computation. However, this design suffers from data inefficiency, group synchronization barriers, and inflexibility with structured rollouts. In this work, we revisit the role of the ``group'' and show that its underlying function is not merely to estimate baselines but to prevent false penalties on negative samples. Building on this insight, we propose negative token filtering, a simple and effective strategy that enables stable single-rollout training. We apply it to two batch-level advantage methods, achieving comparable performance on reasoning tasks and stronger performance on agentic tasks relative to group-based RL techniques.