Automatic Neural Network Pruning that Efficiently Preserves the Model Accuracy
This addresses the issue of high computational resource usage in neural networks for applications with limited hardware, offering an efficient pruning solution with incremental improvements over existing methods.
The paper tackles the problem of neural network pruning to reduce computational costs (FLOPs) while preserving accuracy, proposing an automatic method that learns which neurons to prune with minimal finetuning. It achieves a 52.00% FLOPs reduction on ResNet-50 with a state-of-the-art accuracy of 76.63% after finetuning on ILSVRC2012.
Neural networks performance has been significantly improved in the last few years, at the cost of an increasing number of floating point operations per second (FLOPs). However, more FLOPs can be an issue when computational resources are limited. As an attempt to solve this problem, pruning filters is a common solution, but most existing pruning methods do not preserve the model accuracy efficiently and therefore require a large number of finetuning epochs. In this paper, we propose an automatic pruning method that learns which neurons to preserve in order to maintain the model accuracy while reducing the FLOPs to a predefined target. To accomplish this task, we introduce a trainable bottleneck that only requires one single epoch with 25.6% (CIFAR-10) or 7.49% (ILSVRC2012) of the dataset to learn which filters to prune. Experiments on various architectures and datasets show that the proposed method can not only preserve the accuracy after pruning but also outperform existing methods after finetuning. We achieve a 52.00% FLOPs reduction on ResNet-50, with a Top-1 accuracy of 47.51% after pruning and a state-of-the-art (SOTA) accuracy of 76.63% after finetuning on ILSVRC2012. Code available at https://github.com/nota-github/autobot_AAAI23.