A New Training Framework for Deep Neural Network
This addresses the problem of model deployment efficiency for practitioners by offering an incremental improvement over existing distillation methods.
The paper tackles the computational and storage overhead of requiring pre-trained teacher models in knowledge distillation by proposing a novel Self Distillation (SD) framework, demonstrating its effectiveness with performance improvements across diverse tasks and datasets.
Knowledge distillation is the process of transferring the knowledge from a large model to a small model. In this process, the small model learns the generalization ability of the large model and retains the performance close to that of the large model. Knowledge distillation provides a training means to migrate the knowledge of models, facilitating model deployment and speeding up inference. However, previous distillation methods require pre-trained teacher models, which still bring computational and storage overheads. In this paper, a novel general training framework called Self Distillation (SD) is proposed. We demonstrate the effectiveness of our method by enumerating its performance improvements in diverse tasks and benchmark datasets.