RL for Reasoning by Adaptively Revealing Rationales
This work addresses the problem of inefficient training for long-sequence tasks in AI, offering a novel approach that bridges supervised and reinforcement learning, though it is incremental in combining curriculum learning with existing methods.
The paper tackles the challenge of training models on complex sequence generation tasks by introducing adaptive backtracking (AdaBack), a per-sample curriculum learning algorithm that dynamically reveals partial target outputs based on past reward signals, enabling models to solve problems with long latent dependencies that supervised fine-tuning and reinforcement learning alone fail on, achieving reliable performance on synthetic tasks and mathematical reasoning benchmarks like MATH and GSM8k.
We propose that reinforcement learning (RL) from partial expert demonstrations is not merely a training heuristic, but a promising framework for solving complex sequence generation tasks. Supervised fine-tuning (SFT) relies on dense ground-truth labels, which become increasingly costly as sequence length grows. RL, on the other hand, struggles with sparse rewards and a combinatorially large output space. We address this by introducing adaptive backtracking (AdaBack), a per-sample curriculum learning algorithm that reveals only a partial prefix of the target output during training. The supervision length is adjusted dynamically for each sample based on the model's past reward signal, allowing it to incrementally learn to complete reasoning chains by conditioning on correct partial solutions. We investigate this intermediate regime between SFT and RL and argue that per-sample curriculum learning is more than a trade-off between efficiency and generality, it can succeed in tasks with long sequences of latent dependencies where SFT and RL both fail to generalize. Using a synthetic task with latent parity constraints, we show that our adaptive curriculum over partial answers reliably solves problems that are otherwise intractable. On mathematical reasoning benchmarks (MATH, GSM8k), we find that curriculum learning enables models to solve problems that RL alone cannot, acquiring new reasoning capabilities through incremental exposure to partial solutions.