Bayesian Sparsification of Recurrent Neural Networks
This reduces memory usage for RNNs in text analysis tasks, but it is incremental as it extends an existing sparsification method to RNNs.
The paper tackles the problem of memory-intensive weights in recurrent neural networks (RNNs) by applying Sparse Variational Dropout to sparsify them, achieving 99.5% sparsity on sentiment analysis without quality loss and up to 87% sparsity on language modeling with slight accuracy loss.
Recurrent neural networks show state-of-the-art results in many text analysis tasks but often require a lot of memory to store their weights. Recently proposed Sparse Variational Dropout eliminates the majority of the weights in a feed-forward neural network without significant loss of quality. We apply this technique to sparsify recurrent neural networks. To account for recurrent specifics we also rely on Binary Variational Dropout for RNN. We report 99.5% sparsity level on sentiment analysis task without a quality drop and up to 87% sparsity level on language modeling task with slight loss of accuracy.