SLIM LSTMs
This work addresses efficiency issues in LSTM networks for machine learning practitioners, but it is incremental as it builds on existing LSTM structures.
The paper tackled the problem of redundancy and high parameterization in standard LSTM RNNs by introducing SLIM LSTM variants, which aggressively reduce parameters to achieve computational savings and speedup in training while maintaining validation accuracy comparable to standard LSTMs.
Long Short-Term Memory (LSTM) Recurrent Neural networks (RNNs) rely on gating signals, each driven by a function of a weighted sum of at least 3 components: (i) one of an adaptive weight matrix multiplied by the incoming external input vector sequence, (ii) one adaptive weight matrix multiplied by the previous memory/state vector, and (iii) one adaptive bias vector. In effect, they augment the simple Recurrent Neural Networks (sRNNs) structure with the addition of a "memory cell" and the incorporation of at most 3 gating signals. The standard LSTM structure and components encompass redundancy and overly increased parameterization. In this paper, we systemically introduce variants of the LSTM RNNs, referred to as SLIM LSTMs. These variants express aggressively reduced parameterizations to achieve computational saving and/or speedup in (training) performance---while necessarily retaining (validation accuracy) performance comparable to the standard LSTM RNN.