Exploiting Large-scale Teacher-Student Training for On-device Acoustic Models
This work addresses efficiency and performance challenges for on-device speech recognition systems, though it appears incremental in its approach to semi-supervised learning.
The paper tackles the problem of training small acoustic models for on-device use by exploiting large-scale semi-supervised learning with teacher-student training, achieving up to 14.4% word error rate reduction compared to supervised baselines.
We present results from Alexa speech teams on semi-supervised learning (SSL) of acoustic models (AM) with experiments spanning over 3000 hours of GPU time, making our study one of the largest of its kind. We discuss SSL for AMs in a small footprint setting, showing that a smaller capacity model trained with 1 million hours of unsupervised data can outperform a baseline supervised system by 14.3% word error rate reduction (WERR). When increasing the supervised data to seven-fold, our gains diminish to 7.1% WERR; to improve SSL efficiency at larger supervised data regimes, we employ a step-wise distillation into a smaller model, obtaining a WERR of 14.4%. We then switch to SSL using larger student models in low data regimes; while learning efficiency with unsupervised data is higher, student models may outperform teacher models in such a setting. We develop a theoretical sketch to explain this behavior.