Attention-Based Guided Structured Sparsity of Deep Neural Networks
This work addresses the problem of accuracy loss in neural network pruning for researchers and practitioners in machine learning, offering an incremental improvement over existing methods.
The paper tackles the problem of uncontrolled sparsity in network pruning, which can cause severe accuracy drops, by proposing an attention mechanism that controls sparsity intensity and supervises pruning to preserve important information bottlenecks. On CIFAR-10, the method outperforms the best baseline by 6% and reduces the accuracy drop by 2.6x at the same sparsity level.
Network pruning is aimed at imposing sparsity in a neural network architecture by increasing the portion of zero-valued weights for reducing its size regarding energy-efficiency consideration and increasing evaluation speed. In most of the conducted research efforts, the sparsity is enforced for network pruning without any attention to the internal network characteristics such as unbalanced outputs of the neurons or more specifically the distribution of the weights and outputs of the neurons. That may cause severe accuracy drop due to uncontrolled sparsity. In this work, we propose an attention mechanism that simultaneously controls the sparsity intensity and supervised network pruning by keeping important information bottlenecks of the network to be active. On CIFAR-10, the proposed method outperforms the best baseline method by 6% and reduced the accuracy drop by 2.6x at the same level of sparsity.