F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare
This addresses a bottleneck in RLVR for AI alignment and code generation by improving performance on rare-correct modes, though it is an incremental modification to existing methods.
The paper tackles the problem of reinforcement learning with verifiable rewards (RLVR) being biased toward common solutions due to computational limits on group sizes, which miss rare-correct trajectories. It proposes a difficulty-aware advantage scaling coefficient that improves pass@256 scores on benchmarks, such as from 64.1 to 70.3 for GRPO, without increasing computational cost.
Reinforcement Learning with Verifiable Rewards (RLVR) is commonly based on group sampling to estimate advantages and stabilize policy updates. In practice, large group sizes are not feasible due to computational limits, which biases learning toward trajectories that are already likely. Smaller groups often miss rare-correct trajectories while still containing mixed rewards, concentrating probability on common solutions. We derive the probability that updates miss rare-correct modes as a function of group size, showing non-monotonic behavior, and characterize how updates redistribute mass within the correct set, revealing that unsampled-correct mass can shrink even as total correct mass grows. Motivated by this analysis, we propose a difficulty-aware advantage scaling coefficient, inspired by Focal loss, that down-weights updates on high-success prompts. The lightweight modification can be directly integrated into any group-relative RLVR algorithm such as GRPO, DAPO, and CISPO. On Qwen2.5-7B across in-domain and out-of-domain benchmarks, our method improves pass@256 from 64.1 $\rightarrow$ 70.3 (GRPO), 69.3 $\rightarrow$ 72.5 (DAPO), and 73.2 $\rightarrow$ 76.8 (CISPO), while preserving or improving pass@1, without increasing group size or computational cost.