Exploring Sparsity in Recurrent Neural Networks
This addresses deployment challenges for RNNs on resource-constrained devices like mobile phones and embedded systems, offering an incremental improvement in efficiency.
The paper tackles the problem of large model sizes and slow evaluation times in Recurrent Neural Networks (RNNs) by proposing a pruning technique during initial training, resulting in up to 90% model size reduction and 2x to 7x speed-up while maintaining accuracy close to the original dense network.
Recurrent Neural Networks (RNN) are widely used to solve a variety of problems and as the quantity of data and the amount of available compute have increased, so have model sizes. The number of parameters in recent state-of-the-art networks makes them hard to deploy, especially on mobile phones and embedded devices. The challenge is due to both the size of the model and the time it takes to evaluate it. In order to deploy these RNNs efficiently, we propose a technique to reduce the parameters of a network by pruning weights during the initial training of the network. At the end of training, the parameters of the network are sparse while accuracy is still close to the original dense neural network. The network size is reduced by 8x and the time required to train the model remains constant. Additionally, we can prune a larger dense network to achieve better than baseline performance while still reducing the total number of parameters significantly. Pruning RNNs reduces the size of the model and can also help achieve significant inference time speed-up using sparse matrix multiply. Benchmarks show that using our technique model size can be reduced by 90% and speed-up is around 2x to 7x.