Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data
For LLM reasoning, this addresses a critical failure mode in RL training that arises with strong base models, offering a practical solution to maintain diversity and improve generalization.
Reinforcement learning on LLMs suffers from mode collapse when base models saturate benchmarks with correct but homogeneous solutions. The proposed Mixed-CUTS framework prevents this and boosts AIME25 Pass@1 accuracy by up to 15.1% over standard GRPO.
Reinforcement Learning (RL) enhances LLM reasoning, yet a paradox emerges as models scale: strong base models saturate standard benchmarks (e.g., MATH), yielding correct but homogeneous solutions. In such environments, the lack of failure cases causes the advantage signal in group-relative algorithms (e.g., GRPO) to vanish, driving policies into mode collapse. To address this, we propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration. Unlike standard sampling that follows model biases, CUTS flattens the local optimization landscape by sampling uniformly from constrained high-confidence candidates. We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance. Experiments on Qwen3 models demonstrate that our approach prevents policy degeneration and significantly boosts out-of-domain generalization. Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO, validating that maintaining diversity within the semantic manifold is critical for rigorous reasoning.