Predefined Sparseness in Recurrent Sequence Models
This addresses memory efficiency for NLP practitioners, but it is incremental as it builds on existing sparseness techniques.
The paper tackles the problem of reducing memory footprint during training of recurrent sequence models by enforcing sparseness upfront, showing that predefined sparseness in language modeling increases hidden state sizes without extra parameters and in sequence labeling achieves similar performance with fewer trainable parameters.
Inducing sparseness while training neural networks has been shown to yield models with a lower memory footprint but similar effectiveness to dense models. However, sparseness is typically induced starting from a dense model, and thus this advantage does not hold during training. We propose techniques to enforce sparseness upfront in recurrent sequence models for NLP applications, to also benefit training. First, in language modeling, we show how to increase hidden state sizes in recurrent layers without increasing the number of parameters, leading to more expressive models. Second, for sequence labeling, we show that word embeddings with predefined sparseness lead to similar performance as dense embeddings, at a fraction of the number of trainable parameters.