On the Position Bias of On-Policy Distillation
Improves learning efficiency and final performance of reinforcement learning with teacher-student distillation, addressing a previously overlooked bias in token-level supervision.
On-Policy Distillation (OPD) suffers from position bias: later tokens in student rollouts have degraded supervision quality. The proposed Importance-Weighted OPD (IW-OPD) reweights tokens based on distribution discrepancy, achieving up to 6.9 points improvement on AIME-2025 and faster convergence.
On-Policy Distillation (OPD) improves the learning efficiency of standard reinforcement learning through dense, token-level supervision from teachers. In the standard KL objective of OPD, token-level losses are uniformly averaged, implying equal weights for all tokens. However, we discover that not all tokens are created equal: as student rollouts grow longer, they deviate further from the teacher's distribution, leading to degraded supervision quality at later positions. As a result, OPD using only the first 30% of tokens can perform comparably to using all tokens, whereas OPD using only the last 30% of tokens barely learns anything. In this work, we provide a principled understanding of this issue through the lens of constrained optimization. Based on these insights, we derive Importance-Weighted On-Policy Distillation (IW-OPD), in which the weight assigned to each token depends on the accumulated discrepancy between the student's and teacher's distributions, naturally upweighting earlier tokens and downweighting later ones with larger deviations. We show that IW-OPD converges significantly faster than OPD, with better learning efficiency, and achieves better final performance than standard OPD in both same-size and cross-scale settings, improving performance up to 6.9 points on AIME-2025.