RESTRAIN: From Spurious Votes to Signals -- Self-Driven RL with Self-Penalization
This work addresses the scalability of reasoning in AI by reducing reliance on expensive labeled data, though it is incremental as it builds on existing policy optimization methods.
The paper tackles the problem of high costs and limitations of human-annotated data in reinforcement learning for reasoning models by introducing RESTRAIN, a self-penalizing RL framework that uses unlabeled data to improve performance, achieving gains such as up to +140.7% Pass@1 on AIME25 and nearly matching gold-label training without supervision.
Reinforcement learning with human-annotated data has boosted chain-of-thought reasoning in large reasoning models, but these gains come at high costs in labeled data while faltering on harder tasks. A natural next step is experience-driven learning, where models improve without curated labels by adapting to unlabeled data. We introduce RESTRAIN (REinforcement learning with Self-restraint), a self-penalizing RL framework that converts the absence of gold labels into a useful learning signal. Instead of overcommitting to spurious majority votes, RESTRAIN exploits signals from the model's entire answer distribution: penalizing overconfident rollouts and low-consistency examples while preserving promising reasoning chains. The self-penalization mechanism integrates seamlessly into policy optimization methods such as GRPO, enabling continual self-improvement without supervision. On challenging reasoning benchmarks, RESTRAIN delivers large gains using only unlabeled data. With Qwen3-4B-Base and OctoThinker Hybrid-8B-Base, it improves Pass@1 by up to +140.7 percent on AIME25, +36.2 percent on MMLU_STEM, and +19.6 percent on GPQA-Diamond, nearly matching gold-label training while using no gold labels. These results demonstrate that RESTRAIN establishes a scalable path toward stronger reasoning without gold labels.