Bayesian Compression for Natural Language Processing
This addresses efficiency and interpretability issues for NLP practitioners using RNNs, though it appears incremental as it builds on existing compression methods.
The paper tackles the problem of large parameter sizes in recurrent neural networks (RNNs) for natural language processing, particularly in embedding layers, by proposing a Bayesian sparsification technique that compresses RNNs dozens to hundreds of times without extensive hyperparameter tuning, and further generalizes it for vocabulary sparsification to filter out unnecessary words.
In natural language processing, a lot of the tasks are successfully solved with recurrent neural networks, but such models have a huge number of parameters. The majority of these parameters are often concentrated in the embedding layer, which size grows proportionally to the vocabulary length. We propose a Bayesian sparsification technique for RNNs which allows compressing the RNN dozens or hundreds of times without time-consuming hyperparameters tuning. We also generalize the model for vocabulary sparsification to filter out unnecessary words and compress the RNN even further. We show that the choice of the kept words is interpretable. Code is available on github: https://github.com/tipt0p/SparseBayesianRNN