Revisiting Self-Distillation
This work addresses the problem of understanding and improving model generalization in machine learning, particularly for practitioners using distillation techniques, but it is incremental as it builds on existing self-distillation research.
The paper systematically studies self-distillation, showing that it consistently allows a student model to outperform a highly accurate teacher, and provides an alternative explanation based on loss landscape geometry, linking it to flatter minima and better generalization.
Knowledge distillation is the procedure of transferring "knowledge" from a large model (the teacher) to a more compact one (the student), often being used in the context of model compression. When both models have the same architecture, this procedure is called self-distillation. Several works have anecdotally shown that a self-distilled student can outperform the teacher on held-out data. In this work, we systematically study self-distillation in a number of settings. We first show that even with a highly accurate teacher, self-distillation allows a student to surpass the teacher in all cases. Secondly, we revisit existing theoretical explanations of (self) distillation and identify contradicting examples, revealing possible drawbacks of these explanations. Finally, we provide an alternative explanation for the dynamics of self-distillation through the lens of loss landscape geometry. We conduct extensive experiments to show that self-distillation leads to flatter minima, thereby resulting in better generalization.