Fast and Effective On-policy Distillation from Reasoning Prefixes
This work addresses the training efficiency problem for researchers and practitioners using on-policy distillation in language models, offering a significant speed-up with minimal performance loss, though it is incremental as it builds on existing distillation techniques.
The paper tackles the high computational cost of on-policy distillation by proposing a method that applies distillation only to prefixes of student-generated outputs and terminates sampling early, achieving performance matching full on-policy distillation while reducing training FLOP by 2x-47x on AI-for-Math and out-of-domain benchmarks.
On-policy distillation (OPD), which samples trajectories from the student model and supervises them with a teacher at the token level, avoids relying solely on verifiable terminal rewards and can yield better generalization than off-policy distillation. However, OPD requires expensive on-the-fly sampling of the student policy during training, which substantially increases training cost, especially for long responses. Our initial analysis shows that, during OPD, training signals are often concentrated in the prefix of each output, and that even a short teacher-generated prefix can significantly help the student produce the correct answer. Motivated by these observations, we propose a simple yet effective modification of OPD: we apply the distillation objective only to prefixes of student-generated outputs and terminate each sampling early during distillation. Experiments on a suite of AI-for-Math and out-of-domain benchmarks show that on-policy prefix distillation matches the performance of full OPD while reducing training FLOP by 2x-47x.