A Framework For Pruning Deep Neural Networks Using Energy-Based Models
This addresses the challenge of network capacity and efficiency for machine learning practitioners, but it is incremental as it builds on existing pruning methods.
The paper tackles the problem of reducing parameters in deep neural networks by proposing a pruning framework using a population-based global optimization method, achieving over 50% pruning with less than 5% and 1% drops in Top-1 and Top-5 accuracy on CIFAR datasets.
A typical deep neural network (DNN) has a large number of trainable parameters. Choosing a network with proper capacity is challenging and generally a larger network with excessive capacity is trained. Pruning is an established approach to reducing the number of parameters in a DNN. In this paper, we propose a framework for pruning DNNs based on a population-based global optimization method. This framework can use any pruning objective function. As a case study, we propose a simple but efficient objective function based on the concept of energy-based models. Our experiments on ResNets, AlexNet, and SqueezeNet for the CIFAR-10 and CIFAR-100 datasets show a pruning rate of more than $50\%$ of the trainable parameters with approximately $<5\%$ and $<1\%$ drop of Top-1 and Top-5 classification accuracy, respectively.