Iterative evaluation of LSTM cells
This is an incremental improvement for language modeling tasks.
The authors tackled the problem of improving LSTM network performance by modifying the information flow to an iterative scheme, which enhanced model performance comparably to increasing parameters by over 3 times.
In this work we present a modification in the conventional flow of information through a LSTM network, which we consider well suited for RNNs in general. The modification leads to a iterative scheme where the computations performed by the LSTM cell are repeated over a constant input and cell state values, while updating the hidden state a finite number of times. We provide theoretical and empirical evidence to support the augmented capabilities of the iterative scheme and show examples related to language modeling. The modification yields an enhancement in the model performance comparable with the original model augmented more than 3 times in terms of the total amount of parameters.