CES-KD: Curriculum-based Expert Selection for Guided Knowledge Distillation
This work addresses model compression for edge devices by enhancing knowledge distillation efficiency, though it appears incremental as it builds on existing teacher assistant methods.
The paper tackles the capacity gap problem in knowledge distillation by proposing CES-KD, a curriculum-based method that selects teachers based on image difficulty, resulting in improved accuracy on datasets like CIFAR-10, CIFAR-100, CINIC-10, and ImageNet for models such as VGG, ResNets, and WideResNets.
Knowledge distillation (KD) is an effective tool for compressing deep classification models for edge devices. However, the performance of KD is affected by the large capacity gap between the teacher and student networks. Recent methods have resorted to a multiple teacher assistant (TA) setting for KD, which sequentially decreases the size of the teacher model to relatively bridge the size gap between these models. This paper proposes a new technique called Curriculum Expert Selection for Knowledge Distillation (CES-KD) to efficiently enhance the learning of a compact student under the capacity gap problem. This technique is built upon the hypothesis that a student network should be guided gradually using stratified teaching curriculum as it learns easy (hard) data samples better and faster from a lower (higher) capacity teacher network. Specifically, our method is a gradual TA-based KD technique that selects a single teacher per input image based on a curriculum driven by the difficulty in classifying the image. In this work, we empirically verify our hypothesis and rigorously experiment with CIFAR-10, CIFAR-100, CINIC-10, and ImageNet datasets and show improved accuracy on VGG-like models, ResNets, and WideResNets architectures.