Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level
For researchers training language models via distillation, AOPD offers a more stable and effective alternative to standard on-policy distillation.
Asymmetric On-Policy Distillation (AOPD) addresses structural weaknesses in standard on-policy distillation for language model training, achieving average gains of 4.09 and 8.34 points on mathematical reasoning benchmarks under strong and weak initialization, respectively.
On-policy distillation (OPD) trains a student on its own trajectories with token-level teacher feedback and often outperforms off-policy distillation and standard reinforcement learning. However, we find that its standard advantage weighted policy gradient suffers from three structural weaknesses, including high variance updates, vanishing gradients in zero-advantage regions, and exploration bottlenecks when corrective signals are insufficient.We therefore propose Asymmetric On-Policy Distillation (AOPD), which replaces ineffective negative reinforcement with localized divergence minimization in non-positive advantage regions while preserving positive reinforcement learning. Experiments on mathematical reasoning benchmarks show that AOPD consistently outperforms standard OPD, with average gains of 4.09 / 8.34 under strong / weak initialization, respectively. AOPD also maintains higher policy entropy during training and better capability retention during sequential tool-use adaptation.