Rethinking the Knowledge Distillation From the Perspective of Model Calibration
This addresses a bottleneck in knowledge distillation for improving model efficiency, but it is incremental as it builds on existing calibration techniques.
The paper tackles the problem that more accurate teacher models do not always lead to better student models in knowledge distillation, finding that larger teachers can be over-confident, and shows that calibrating the teacher model makes teacher size positively correlate with student performance.
Recent years have witnessed dramatically improvements in the knowledge distillation, which can generate a compact student model for better efficiency while retaining the model effectiveness of the teacher model. Previous studies find that: more accurate teachers do not necessary make for better teachers due to the mismatch of abilities. In this paper, we aim to analysis the phenomenon from the perspective of model calibration. We found that the larger teacher model may be too over-confident, thus the student model cannot effectively imitate. While, after the simple model calibration of the teacher model, the size of the teacher model has a positive correlation with the performance of the student model.