SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision
For practitioners of on-policy distillation, SEAD addresses the inefficiency of uniform supervision by dynamically adapting to student competence, yielding significant gains in accuracy.
SEAD introduces entropy-guided supervision for on-policy distillation, skipping ~50% of tokens and achieving +4.8 average accuracy over vanilla OPD on six math benchmarks with OLMo-3 models.
On-policy distillation (OPD) has a property absent in offline distillation and RL: teacher supervision quality depends on student competence. Incoherent rollouts yield noisy gradients; already-mastered tokens yield redundant ones. This creates waste at three scales (tokens, training phases, and prompts) yet existing methods supervise uniformly. We introduce SEAD, which uses entropy as a unified probe of this competence-dependent degradation at three scales: (1) joint teacher-student entropy partitions tokens into zones receiving tailored divergences or zero gradient (approx. 50% skipped); (2) a cosine schedule anneals from forward to reverse KL as competence grows; (3) a competence-gated curriculum introduces prompts easy-to-hard. These components are symbiotically necessary: token selection requires coherent rollouts (curriculum), annealing requires monotonic improvement (also curriculum). On OLMo-3 (7B to 32B), SEAD achieves +4.8 avg accuracy over vanilla OPD across six math benchmarks, with ablations confirming super-additive interactions.