Do Not Blindly Imitate the Teacher: Using Perturbed Loss for Knowledge Distillation
This work addresses a specific bottleneck in knowledge distillation for machine learning practitioners, offering an incremental improvement over existing methods.
The authors tackled the problem of suboptimal student performance in knowledge distillation due to discrepancies between teacher and ground truth distributions, proposing a perturbed loss (PTLoss) that improves distillation effectiveness across five datasets with significant gains.
Knowledge distillation is a popular technique to transfer knowledge from large teacher models to a small student model. Typically, the student learns to imitate the teacher by minimizing the KL divergence of its output distribution with the teacher's output distribution. In this work, we argue that such a learning objective is sub-optimal because there exists a discrepancy between the teacher's output distribution and the ground truth label distribution. Therefore, forcing the student to blindly imitate the unreliable teacher output distribution leads to inferior performance. To this end, we propose a novel knowledge distillation objective PTLoss by first representing the vanilla KL-based distillation loss function via a Maclaurin series and then perturbing the leading-order terms in this series. This perturbed loss implicitly transforms the original teacher into a proxy teacher with a distribution closer to the ground truth distribution. We establish the theoretical connection between this "distribution closeness" and the student model generalizability, which enables us to select the PTLoss's perturbation coefficients in a principled way. Extensive experiments on five datasets demonstrate PTLoss can significantly improve the distillation effectiveness for teachers of various scales.