Efficient Knowledge Distillation from Model Checkpoints
This work addresses the efficiency of knowledge distillation for practitioners by identifying a counterintuitive but effective approach to teacher selection, offering incremental improvements in model compression.
The paper tackles the problem of selecting optimal teachers for knowledge distillation by showing that intermediate model checkpoints, despite lower accuracy, often outperform fully converged models as teachers, and that a weak ensemble of intermediate models can surpass a strong ensemble of independently trained models, with experiments verifying an algorithm that improves distillation effectiveness.
Knowledge distillation is an effective approach to learn compact models (students) with the supervision of large and strong models (teachers). As empirically there exists a strong correlation between the performance of teacher and student models, it is commonly believed that a high performing teacher is preferred. Consequently, practitioners tend to use a well trained network or an ensemble of them as the teacher. In this paper, we make an intriguing observation that an intermediate model, i.e., a checkpoint in the middle of the training procedure, often serves as a better teacher compared to the fully converged model, although the former has much lower accuracy. More surprisingly, a weak snapshot ensemble of several intermediate models from a same training trajectory can outperform a strong ensemble of independently trained and fully converged models, when they are used as teachers. We show that this phenomenon can be partially explained by the information bottleneck principle: the feature representations of intermediate models can have higher mutual information regarding the input, and thus contain more "dark knowledge" for effective distillation. We further propose an optimal intermediate teacher selection algorithm based on maximizing the total task-related mutual information. Experiments verify its effectiveness and applicability.