Revisiting Architecture-aware Knowledge Distillation: Smaller Models and Faster Search
This work addresses the bottleneck of high computational cost in architecture-aware knowledge distillation, offering a faster and more effective method for compressing neural networks, though it appears incremental as it builds on existing distillation and search techniques.
The paper tackles the problem of computationally expensive architecture search in knowledge distillation by introducing TRADE, a trust region Bayesian optimization algorithm that efficiently finds effective student architectures, achieving consistent performance improvements over conventional NAS and pre-defined architectures.
Knowledge Distillation (KD) has recently emerged as a popular method for compressing neural networks. In recent studies, generalized distillation methods that find parameters and architectures of student models at the same time have been proposed. Still, this search method requires a lot of computation to search for architectures and has the disadvantage of considering only convolutional blocks in their search space. This paper introduces a new algorithm, coined as Trust Region Aware architecture search to Distill knowledge Effectively (TRADE), that rapidly finds effective student architectures from several state-of-the-art architectures using trust region Bayesian optimization approach. Experimental results show our proposed TRADE algorithm consistently outperforms both the conventional NAS approach and pre-defined architecture under KD training.