FastWhisper: Adaptive Self-knowledge Distillation for Real-time Automatic Speech Recognition
This addresses model compression for real-time automatic speech recognition, offering incremental improvements in efficiency and accuracy.
The paper tackled the problem of student models inheriting teacher model shortcomings in knowledge distillation, proposing adaptive self-knowledge distillation (ASKD) to improve generalization, resulting in FastWhisper achieving a 1.07% lower word error rate and 5x faster inference time than the teacher model.
Knowledge distillation is one of the most effective methods for model compression. Previous studies have focused on the student model effectively training the predictive distribution of the teacher model. However, during training, the student model may inherit the shortcomings of the teacher model, which can lead to a decline in generalization capacity. To mitigate this issue, we propose adaptive self-knowledge distillation (ASKD), which dynamically reduces the dependence of the teacher model to improve the self-training capacity, and performs the self-knowledge distillation method to improve the generalization capacity of the student model. We further distill the Whisper model into a smaller variant, called FastWhisper. In our post-training setting, FastWhisper achieved a word error rate of 1.07% lower than the teacher model Whisper, and its relative inference time was 5 times faster.