Improving Streaming Automatic Speech Recognition With Non-Streaming Model Distillation On Unsupervised Data
This work addresses the problem of improving accuracy for low-latency streaming ASR models used in smart speakers and on-device applications, representing an incremental advance by applying distillation with unsupervised data.
The authors tackled the performance gap between streaming and non-streaming automatic speech recognition models by using non-streaming models as teachers to generate transcripts on large unsupervised datasets for knowledge distillation into streaming models. They scaled training to 3 million hours of YouTube audio and achieved a 16.4% relative reduction in word error rate for French compared to a baseline streaming model.
Streaming end-to-end automatic speech recognition (ASR) models are widely used on smart speakers and on-device applications. Since these models are expected to transcribe speech with minimal latency, they are constrained to be causal with no future context, compared to their non-streaming counterparts. Consequently, streaming models usually perform worse than non-streaming models. We propose a novel and effective learning method by leveraging a non-streaming ASR model as a teacher to generate transcripts on an arbitrarily large data set, which is then used to distill knowledge into streaming ASR models. This way, we scale the training of streaming models to up to 3 million hours of YouTube audio. Experiments show that our approach can significantly reduce the word error rate (WER) of RNNT models not only on LibriSpeech but also on YouTube data in four languages. For example, in French, we are able to reduce the WER by 16.4% relatively to a baseline streaming model by leveraging a non-streaming teacher model trained on the same amount of labeled data as the baseline.