Diagnosing and Mitigating System Bias in Self-Rewarding RL
This addresses a key bottleneck in scaling reinforcement learning for large language models in unlabeled settings, though it appears incremental as it builds on existing RLIR methods.
The paper tackles the problem of system bias in self-rewarding reinforcement learning (RLIR), where models overestimate high-confidence rollouts, leading to biased and unstable reward estimation. The proposed RLER method improves performance by +13.6% over RLIR and reduces the gap to RLVR to only 3.6%, enabling stable scaling on unlabeled samples.
Reinforcement learning with verifiable rewards (RLVR) scales the reasoning ability of large language models (LLMs) but remains bottlenecked by limited labeled samples for continued data scaling. Reinforcement learning with intrinsic rewards (RLIR), where the policy model assigns rewards to its own rollouts, enables sustainable scaling in unlabeled settings, yet its performance and stability lag behind RLVR. We trace this gap to a system bias: the model tends to overestimate its high-confidence rollouts, leading to biased and unstable reward estimation. This bias accumulates as training progresses, with deviations from the oracle drifting toward over-reward, causing unstable training. We characterize this bias using three metrics: $ρ_{\text{noise}}$, $ρ_{\text{selfbias}}$, and $ρ_{\text{symbias}}$. We find that $ρ_{\text{noise}}$ and $ρ_{\text{symbias}}$ impact convergence, while $ρ_{\text{selfbias}}$ amplifies both correct and incorrect updates, leading to instability. To mitigate this, we propose reinforcement learning with ensembled rewards (RLER), which aggregates diverse models and adapts reward interpolation and rollout selection. Extensive experiments show that RLER improves by +13.6% over RLIR and is only 3.6% below RLVR, achieving stable scaling on unlabeled samples, making it highly applicable.