An Embarrassingly Simple Approach for Knowledge Distillation
This work addresses a practical bottleneck in knowledge distillation for machine learning practitioners by simplifying application across datasets and tasks.
The paper tackles the problem of balancing task and distillation losses in knowledge distillation by proposing a two-stage approach that first transfers backbone knowledge and then learns the task-head, eliminating the need for loss weight tuning. It shows significant performance gains on CIFAR-100 and ImageNet, narrowing the student-teacher gap and outperforming state-of-the-art methods.
Knowledge Distillation (KD) aims at improving the performance of a low-capacity student model by inheriting knowledge from a high-capacity teacher model. Previous KD methods typically train a student by minimizing a task-related loss and the KD loss simultaneously, using a pre-defined loss weight to balance these two terms. In this work, we propose to first transfer the backbone knowledge from a teacher to the student, and then only learn the task-head of the student network. Such a decomposition of the training process circumvents the need of choosing an appropriate loss weight, which is often difficult in practice, and thus makes it easier to apply to different datasets and tasks. Importantly, the decomposition permits the core of our method, Stage-by-Stage Knowledge Distillation (SSKD), which facilitates progressive feature mimicking from teacher to student. Extensive experiments on CIFAR-100 and ImageNet suggest that SSKD significantly narrows down the performance gap between student and teacher, outperforming state-of-the-art approaches. We also demonstrate the generalization ability of SSKD on other challenging benchmarks, including face recognition on IJB-A dataset as well as object detection on COCO dataset.