DistPro: Searching A Fast Knowledge Distillation Process via Meta Optimization
This work addresses the need for efficient knowledge distillation in deep learning, offering a novel automated approach that can generalize across tasks and networks, though it is incremental in optimizing existing distillation methods.
The paper tackles the problem of automatically searching for an optimal knowledge distillation scheme to improve student network accuracy, especially with faster training, by proposing DistPro, a framework that uses differentiable meta-learning to learn stochastic weighting processes for distillation pathways, achieving state-of-the-art results on CIFAR100 and ImageNet datasets.
Recent Knowledge distillation (KD) studies show that different manually designed schemes impact the learned results significantly. Yet, in KD, automatically searching an optimal distillation scheme has not yet been well explored. In this paper, we propose DistPro, a novel framework which searches for an optimal KD process via differentiable meta-learning. Specifically, given a pair of student and teacher networks, DistPro first sets up a rich set of KD connection from the transmitting layers of the teacher to the receiving layers of the student, and in the meanwhile, various transforms are also proposed for comparing feature maps along its pathway for the distillation. Then, each combination of a connection and a transform choice (pathway) is associated with a stochastic weighting process which indicates its importance at every step during the distillation. In the searching stage, the process can be effectively learned through our proposed bi-level meta-optimization strategy. In the distillation stage, DistPro adopts the learned processes for knowledge distillation, which significantly improves the student accuracy especially when faster training is required. Lastly, we find the learned processes can be generalized between similar tasks and networks. In our experiments, DistPro produces state-of-the-art (SoTA) accuracy under varying number of learning epochs on popular datasets, i.e. CIFAR100 and ImageNet, which demonstrate the effectiveness of our framework.