SuperRL: Reinforcement Learning with Supervision to Boost Language Model Reasoning
This addresses a bottleneck in applying RL to complex reasoning tasks for AI systems, though it is incremental as it builds on existing RL and SFT methods.
The paper tackles the problem of inefficient learning in reinforcement learning for language model reasoning under sparse rewards by introducing SuperRL, a framework that alternates between RL and supervised fine-tuning, resulting in higher sample efficiency, stronger generalization, and improved robustness across diverse benchmarks.
Large language models are increasingly used for complex reasoning tasks where high-quality offline data such as expert-annotated solutions and distilled reasoning traces are often available. However, in environments with sparse rewards, reinforcement learning struggles to sample successful trajectories, leading to inefficient learning. At the same time, these offline trajectories that represent correct reasoning paths are not utilized by standard on-policy reinforcement learning methods. We introduce SuperRL, a unified training framework that adaptively alternates between RL and SFT. Whenever every rollout for a given instance receives zero reward, indicating the absence of a learning signal, SuperRL falls back to SFT on the curated offline data. Extensive experiments across diverse reasoning benchmarks show that SuperRL surpasses vanilla RL by delivering higher sample efficiency, stronger generalization, and improved robustness under sparse rewards.