Factorization tricks for LSTM networks
This work addresses efficiency issues in training large LSTM networks, which is incremental as it builds on existing LSTM methods.
The authors tackled the problem of reducing parameters and accelerating training in large LSTM networks by introducing two factorization tricks, achieving near state-of-the-art perplexity with significantly fewer parameters and faster training.
We present two simple ways of reducing the number of parameters and accelerating the training of large Long Short-Term Memory (LSTM) networks: the first one is "matrix factorization by design" of LSTM matrix into the product of two smaller matrices, and the second one is partitioning of LSTM matrix, its inputs and states into the independent groups. Both approaches allow us to train large LSTM networks significantly faster to the near state-of the art perplexity while using significantly less RNN parameters.