The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
This addresses a critical problem for researchers and practitioners in AI by identifying and mitigating a paradoxical limitation in RLVR for enhancing reasoning in language models, though it is incremental as it builds on existing RLVR methods.
The paper investigates how Reinforcement Learning with Verifiable Rewards (RLVR) can shrink the reasoning boundary in Large Language Models, revealing negative interference and winner-take-all phenomena that reduce Pass@k performance, and proposes a data curation algorithm that improves Pass@k performance by focusing on low-likelihood problems.
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a key method for improving Large Language Models' reasoning capabilities, yet recent evidence suggests it may paradoxically shrink the reasoning boundary rather than expand it. This paper investigates the shrinkage issue of RLVR by analyzing its learning dynamics and reveals two critical phenomena that explain this failure. First, we expose negative interference in RLVR, where learning to solve certain training problems actively reduces the likelihood of correct solutions for others, leading to the decline of Pass@$k$ performance, or the probability of generating a correct solution within $k$ attempts. Second, we uncover the winner-take-all phenomenon: RLVR disproportionately reinforces problems with high likelihood, correct solutions, under the base model, while suppressing other initially low-likelihood ones. Through extensive theoretical and empirical analysis on multiple mathematical reasoning benchmarks, we show that this effect arises from the inherent on-policy sampling in standard RL objectives, causing the model to converge toward narrow solution strategies. Based on these insights, we propose a simple yet effective data curation algorithm that focuses RLVR learning on low-likelihood problems, achieving notable improvement in Pass@$k$ performance. Our code is available at https://github.com/mail-research/SELF-llm-interference.