Neural network relief: a pruning algorithm based on neural activity
This addresses the problem of inefficient neural network inference for AI practitioners by offering a pruning method that reduces parameters while maintaining performance, though it is incremental as it builds on existing pruning strategies.
The paper tackles overparameterization in deep neural networks by proposing an iterative pruning algorithm that deactivates unimportant connections to find simpler subnetworks, achieving comparable accuracy on MNIST with LeNet and higher parameter compression than state-of-the-art methods on VGG and ResNet for CIFAR-10/100 and Tiny-ImageNet.
Current deep neural networks (DNNs) are overparameterized and use most of their neuronal connections during inference for each task. The human brain, however, developed specialized regions for different tasks and performs inference with a small fraction of its neuronal connections. We propose an iterative pruning strategy introducing a simple importance-score metric that deactivates unimportant connections, tackling overparameterization in DNNs and modulating the firing patterns. The aim is to find the smallest number of connections that is still capable of solving a given task with comparable accuracy, i.e. a simpler subnetwork. We achieve comparable performance for LeNet architectures on MNIST, and significantly higher parameter compression than state-of-the-art algorithms for VGG and ResNet architectures on CIFAR-10/100 and Tiny-ImageNet. Our approach also performs well for the two different optimizers considered -- Adam and SGD. The algorithm is not designed to minimize FLOPs when considering current hardware and software implementations, although it performs reasonably when compared to the state of the art.