Are Full Rollouts Necessary for On-Policy Distillation?
This work offers a practical improvement in training efficiency for researchers and practitioners using On-Policy Distillation in long-horizon reasoning tasks.
This paper addresses the computational expense and potential unreliability of full rollouts in On-Policy Distillation (OPD) for long-horizon reasoning. They propose Progressive OPD (POPD) and Truncated OPD (TOPD) as horizon-control strategies, with POPD improving training efficiency by up to 3x and TOPD matching performance with only 10% of the rollout horizon on mathematical reasoning tasks.
On-policy distillation (OPD) provides dense teacher feedback along rollouts generated by the student and has emerged as a promising post-training paradigm for long-horizon reasoning. However, standard OPD typically generates full rollouts during training, which is computationally expensive and may expose the student to unreliable teacher feedback at late rollout positions, especially during early training. We identify the rollout horizon as a key bottleneck in OPD that substantially impacts training efficiency. Unlike Reinforcement Learning with Verifiable Rewards (RLVR), OPD does not require a complete trajectory or a final answer reward to provide learning signals. This observation suggests that full rollouts may not always be necessary for effective OPD. Motivated by this insight, we propose two simple horizon-control strategies: Progressive OPD (POPD), which gradually expands the rollout horizon during training, and Truncated OPD (TOPD), which permanently performs distillation on reliable truncated rollouts. Experiments on mathematical reasoning show that POPD improves the training efficiency of OPD by up to 3$\times$, while TOPD matches OPD performance using only 10\% of the rollout horizon, leading to substantial wall-clock and memory reductions. These results demonstrate that controlling the rollout horizon offers a simple and practical path to more efficient OPD.