The Role of Teacher Calibration in Knowledge Distillation
This work addresses a key bottleneck in model compression for deep learning practitioners, though it is incremental as it builds on existing Knowledge Distillation techniques.
The paper tackles the problem of understanding factors that improve student performance in Knowledge Distillation by revealing a strong correlation between teacher calibration error and student accuracy, and demonstrates that using a calibration method to reduce this error consistently enhances performance across tasks like classification and detection.
Knowledge Distillation (KD) has emerged as an effective model compression technique in deep learning, enabling the transfer of knowledge from a large teacher model to a compact student model. While KD has demonstrated significant success, it is not yet fully understood which factors contribute to improving the student's performance. In this paper, we reveal a strong correlation between the teacher's calibration error and the student's accuracy. Therefore, we claim that the calibration of the teacher model is an important factor for effective KD. Furthermore, we demonstrate that the performance of KD can be improved by simply employing a calibration method that reduces the teacher's calibration error. Our algorithm is versatile, demonstrating effectiveness across various tasks from classification to detection. Moreover, it can be easily integrated with existing state-of-the-art methods, consistently achieving superior performance.