Growing Networks with Autonomous Pruning
This addresses the challenge of parameter efficiency in deep learning for image classification, offering a method to reduce computational costs while maintaining performance, though it is incremental as it builds on existing pruning and growth techniques.
The paper tackles the problem of training sparse neural networks for image classification by introducing Growing Networks with Autonomous Pruning (GNAP), which dynamically adjusts network size and parameters during training to balance accuracy and parameter efficiency, achieving high accuracy with few parameters, such as 99.44% on MNIST with 6.2k parameters and 92.2% on CIFAR10 with 157.8k parameters.
This paper introduces Growing Networks with Autonomous Pruning (GNAP) for image classification. Unlike traditional convolutional neural networks, GNAP change their size, as well as the number of parameters they are using, during training, in order to best fit the data while trying to use as few parameters as possible. This is achieved through two complementary mechanisms: growth and pruning. GNAP start with few parameters, but their size is expanded periodically during training to add more expressive power each time the network has converged to a saturation point. Between these growing phases, model parameters are trained for classification and pruned simultaneously, with complete autonomy by gradient descent. Growing phases allow GNAP to improve their classification performance, while autonomous pruning allows them to keep as few parameters as possible. Experimental results on several image classification benchmarks show that our approach can train extremely sparse neural networks with high accuracy. For example, on MNIST, we achieved 99.44% accuracy with as few as 6.2k parameters, while on CIFAR10, we achieved 92.2\ accuracy with 157.8k parameters.