SGAD: Soft-Guided Adaptively-Dropped Neural Network
This addresses the need for efficient DNNs in resource-constrained applications, but it is incremental as it builds on existing adaptive dropping mechanisms.
The paper tackled the problem of deep neural network compression by proposing a soft-guided adaptively-dropped (SGAD) neural network that adapts to input sample difficulty, reducing FLOPs by 77% with less than 1% accuracy drop on CIFAR-10 compared to a 32-layer residual network.
Deep neural networks (DNNs) have been proven to have many redundancies. Hence, many efforts have been made to compress DNNs. However, the existing model compression methods treat all the input samples equally while ignoring the fact that the difficulties of various input samples being correctly classified are different. To address this problem, DNNs with adaptive dropping mechanism are well explored in this work. To inform the DNNs how difficult the input samples can be classified, a guideline that contains the information of input samples is introduced to improve the performance. Based on the developed guideline and adaptive dropping mechanism, an innovative soft-guided adaptively-dropped (SGAD) neural network is proposed in this paper. Compared with the 32 layers residual neural networks, the presented SGAD can reduce the FLOPs by 77% with less than 1% drop in accuracy on CIFAR-10.