Network Pruning via Annealing and Direct Sparsity Control
This addresses efficiency issues for deploying deep neural networks in resource-constrained environments, but it is an incremental improvement over existing pruning techniques.
The paper tackles the problem of high computational and memory costs in deep neural networks by proposing a novel pruning method that uses an L0 constraint for direct sparsity control, achieving better or competitive performance compared to state-of-the-art methods on synthetic and real vision datasets.
Artificial neural networks (ANNs) especially deep convolutional networks are very popular these days and have been proved to successfully offer quite reliable solutions to many vision problems. However, the use of deep neural networks is widely impeded by their intensive computational and memory cost. In this paper, we propose a novel efficient network pruning method that is suitable for both non-structured and structured channel-level pruning. Our proposed method tightens a sparsity constraint by gradually removing network parameters or filter channels based on a criterion and a schedule. The attractive fact that the network size keeps dropping throughout the iterations makes it suitable for the pruning of any untrained or pre-trained network. Because our method uses a $L_0$ constraint instead of the $L_1$ penalty, it does not introduce any bias in the training parameters or filter channels. Furthermore, the $L_0$ constraint makes it easy to directly specify the desired sparsity level during the network pruning process. Finally, experimental validation on extensive synthetic and real vision datasets show that the proposed method obtains better or competitive performance compared to other states of art network pruning methods.