Distilling Knowledge Using Parallel Data for Far-field Speech Recognition
It addresses performance degradation in far-field speech recognition for applications like smart devices, but it is incremental as it applies an existing knowledge distillation method to a specific domain.
This paper tackled the problem of improving far-field speech recognition by distilling knowledge from a close-talking teacher model to a far-field student model using parallel data, resulting in up to 4.7% absolute word error rate reduction compared to baseline models.
In order to improve the performance for far-field speech recognition, this paper proposes to distill knowledge from the close-talking model to the far-field model using parallel data. The close-talking model is called the teacher model. The far-field model is called the student model. The student model is trained to imitate the output distributions of the teacher model. This constraint can be realized by minimizing the Kullback-Leibler (KL) divergence between the output distribution of the student model and the teacher model. Experimental results on AMI corpus show that the best student model achieves up to 4.7% absolute word error rate (WER) reduction when compared with the conventionally-trained baseline models.