Fixing the Teacher-Student Knowledge Discrepancy in Distillation
This addresses a bottleneck in knowledge distillation for model compression, offering a flexible solution that can be combined with other methods, though it is incremental as it builds on existing distillation frameworks.
The paper tackles the teacher-student knowledge discrepancy problem in knowledge distillation, where mismatched feature representations hinder student learning, and proposes a student-dependent distillation method that improves performance across datasets like CIFAR100, ImageNet, and COCO.
Training a small student network with the guidance of a larger teacher network is an effective way to promote the performance of the student. Despite the different types, the guided knowledge used to distill is always kept unchanged for different teacher and student pairs in previous knowledge distillation methods. However, we find that teacher and student models with different networks or trained from different initialization could have distinct feature representations among different channels. (e.g. the high activated channel for different categories). We name this incongruous representation of channels as teacher-student knowledge discrepancy in the distillation process. Ignoring the knowledge discrepancy problem of teacher and student models will make the learning of student from teacher more difficult. To solve this problem, in this paper, we propose a novel student-dependent distillation method, knowledge consistent distillation, which makes teacher's knowledge more consistent with the student and provides the best suitable knowledge to different student networks for distillation. Extensive experiments on different datasets (CIFAR100, ImageNet, COCO) and tasks (image classification, object detection) reveal the widely existing knowledge discrepancy problem between teachers and students and demonstrate the effectiveness of our proposed method. Our method is very flexible that can be easily combined with other state-of-the-art approaches.