Efficient Knowledge Distillation for RNN-Transducer Models
This work addresses model compression for on-device ASR applications, offering incremental improvements in accuracy for sparse models.
The paper tackles the problem of improving the accuracy of sparse RNN-Transducer models for streaming speech recognition by proposing an efficient knowledge distillation method using only 'y' and 'blank' posterior probabilities, resulting in WER reductions of up to 12.1% on noisy FarField and 4.8% on LibriSpeech test-other datasets.
Knowledge Distillation is an effective method of transferring knowledge from a large model to a smaller model. Distillation can be viewed as a type of model compression, and has played an important role for on-device ASR applications. In this paper, we develop a distillation method for RNN-Transducer (RNN-T) models, a popular end-to-end neural network architecture for streaming speech recognition. Our proposed distillation loss is simple and efficient, and uses only the "y" and "blank" posterior probabilities from the RNN-T output probability lattice. We study the effectiveness of the proposed approach in improving the accuracy of sparse RNN-T models obtained by gradually pruning a larger uncompressed model, which also serves as the teacher during distillation. With distillation of 60% and 90% sparse multi-domain RNN-T models, we obtain WER reductions of 4.3% and 12.1% respectively, on a noisy FarField eval set. We also present results of experiments on LibriSpeech, where the introduction of the distillation loss yields a 4.8% relative WER reduction on the test-other dataset for a small Conformer model.