Distilling the Knowledge of BERT for CTC-based ASR
This work addresses the trade-off between accuracy and inference speed in speech recognition for applications requiring real-time processing, but it is incremental as it extends previous distillation methods from attention-based to CTC-based models.
The paper tackles the problem of slow inference in language model integration for CTC-based automatic speech recognition by proposing a method to distill BERT's knowledge into the model during training, maintaining fast inference while improving accuracy, with experimental results on CSJ and TED-LIUM2 showing performance gains without speed cost.
Connectionist temporal classification (CTC) -based models are attractive because of their fast inference in automatic speech recognition (ASR). Language model (LM) integration approaches such as shallow fusion and rescoring can improve the recognition accuracy of CTC-based ASR by taking advantage of the knowledge in text corpora. However, they significantly slow down the inference of CTC. In this study, we propose to distill the knowledge of BERT for CTC-based ASR, extending our previous study for attention-based ASR. CTC-based ASR learns the knowledge of BERT during training and does not use BERT during testing, which maintains the fast inference of CTC. Different from attention-based models, CTC-based models make frame-level predictions, so they need to be aligned with token-level predictions of BERT for distillation. We propose to obtain alignments by calculating the most plausible CTC paths. Experimental evaluations on the Corpus of Spontaneous Japanese (CSJ) and TED-LIUM2 show that our method improves the performance of CTC-based ASR without the cost of inference speed.