Data-Efficient On-Policy Distillation for Automatic Speech Recognition
For practitioners building compact ASR models, this work shows that on-policy distillation can substantially close the performance gap with larger models while requiring far less supervised data.
The authors propose an on-policy distillation recipe for ASR that improves a 0.6B-parameter model over supervised fine-tuning and outperforms the same-scale Qwen3-ASR-0.6B baseline on four of five benchmarks, using only 100k hours of speech compared to 20M hours for the teacher.
Building competitive automatic speech recognition (ASR) models usually requires large-scale au- dio supervision, which makes reproduction and specialization expensive. We study Ark-ASR, a 0.6B- parameter audio-conditioned language model trained with 100k hours of speech, and examine whether a strong Qwen-ASR teacher can transfer additional recognition capability through on-policy distillation. Across Mandarin and English ASR benchmarks, the proposed training recipe consistently improves over supervised fine-tuning alone and outperforms the same-scale Qwen3-ASR-0.6B baseline on four of five evaluation sets. This is achieved with only 100k hours of speech, compared with the 20M hours of super- vised audio reported for the Qwen3-Omni AuT encoder. The larger Qwen3-ASR-1.7B remains stronger, but the results show that teacher-guided on-policy training can substantially close the gap for compact ASR models under a much smaller audio budget. A support-overlap diagnostic further suggests that the teacher-data stage improves local student-teacher compatibility, matching recent analyses of when on-policy distillation is effective.