ERDE: Entropy-Regularized Distillation for Early-exit
This work addresses efficiency challenges for real-time and edge applications in image classification, but it is incremental as it builds on existing techniques like early exits and knowledge distillation.
The paper tackles the problem of high computational costs in deep neural networks for image classification by proposing ERDE, a method that integrates early exits and knowledge distillation with an entropy-based loss for incorrect teacher classifications, achieving significant reductions in computational complexity without compromising accuracy on datasets like CIFAR10, CIFAR100, and SVHN.
Although deep neural networks and in particular Convolutional Neural Networks have demonstrated state-of-the-art performance in image classification with relatively high efficiency, they still exhibit high computational costs, often rendering them impractical for real-time and edge applications. Therefore, a multitude of compression techniques have been developed to reduce these costs while maintaining accuracy. In addition, dynamic architectures have been introduced to modulate the level of compression at execution time, which is a desirable property in many resource-limited application scenarios. The proposed method effectively integrates two well-established optimization techniques: early exits and knowledge distillation, where a reduced student early-exit model is trained from a more complex teacher early-exit model. The primary contribution of this research lies in the approach for training the student early-exit model. In comparison to the conventional Knowledge Distillation loss, our approach incorporates a new entropy-based loss for images where the teacher's classification was incorrect. The proposed method optimizes the trade-off between accuracy and efficiency, thereby achieving significant reductions in computational complexity without compromising classification performance. The validity of this approach is substantiated by experimental results on image classification datasets CIFAR10, CIFAR100 and SVHN, which further opens new research perspectives for Knowledge Distillation in other contexts.