Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
This addresses the need for efficient deep architectures to reduce resource usage in neural networks, though it is incremental as it builds on existing pruning techniques.
The paper tackles the problem of reducing computational and memory costs in deep neural networks by introducing network trimming, an iterative pruning method that removes neurons with mostly zero activations, achieving high compression ratios without accuracy loss on LeNet and VGG-16.
State-of-the-art neural networks are getting deeper and wider. While their performance increases with the increasing number of layers and neurons, it is crucial to design an efficient deep architecture in order to reduce computational and memory costs. Designing an efficient neural network, however, is labor intensive requiring many experiments, and fine-tunings. In this paper, we introduce network trimming which iteratively optimizes the network by pruning unimportant neurons based on analysis of their outputs on a large dataset. Our algorithm is inspired by an observation that the outputs of a significant portion of neurons in a large network are mostly zero, regardless of what inputs the network received. These zero activation neurons are redundant, and can be removed without affecting the overall accuracy of the network. After pruning the zero activation neurons, we retrain the network using the weights before pruning as initialization. We alternate the pruning and retraining to further reduce zero activations in a network. Our experiments on the LeNet and VGG-16 show that we can achieve high compression ratio of parameters without losing or even achieving higher accuracy than the original network.