Efficient training of lightweight neural networks using Online Self-Acquired Knowledge Distillation
This method addresses the problem of efficient training of lightweight neural networks for practitioners needing faster and more resource-friendly models, though it appears incremental as it builds on existing knowledge distillation techniques.
The paper tackles the computational and memory demands of conventional Knowledge Distillation by proposing Online Self-Acquired Knowledge Distillation (OSAKD), which uses k-nn density estimation to estimate posterior class probabilities as soft labels, and validates its effectiveness on four datasets.
Knowledge Distillation has been established as a highly promising approach for training compact and faster models by transferring knowledge from heavyweight and powerful models. However, KD in its conventional version constitutes an enduring, computationally and memory demanding process. In this paper, Online Self-Acquired Knowledge Distillation (OSAKD) is proposed, aiming to improve the performance of any deep neural model in an online manner. We utilize k-nn non-parametric density estimation technique for estimating the unknown probability distributions of the data samples in the output feature space. This allows us for directly estimating the posterior class probabilities of the data samples, and we use them as soft labels that encode explicit information about the similarities of the data with the classes, negligibly affecting the computational cost. The experimental evaluation on four datasets validates the effectiveness of proposed method.