LSTM with Working Memory
This work addresses the need for more effective yet simple RNN architectures for sequence modeling tasks, though it appears incremental in nature.
The authors tackled the problem of improving upon the widely-used LSTM architecture by proposing a modified LSTM-like design that maintains simplicity while achieving better performance on tested tasks, and they introduced a new RNN benchmark using handwritten digits to stress network capabilities.
Previous RNN architectures have largely been superseded by LSTM, or "Long Short-Term Memory". Since its introduction, there have been many variations on this simple design. However, it is still widely used and we are not aware of a gated-RNN architecture that outperforms LSTM in a broad sense while still being as simple and efficient. In this paper we propose a modified LSTM-like architecture. Our architecture is still simple and achieves better performance on the tasks that we tested on. We also introduce a new RNN performance benchmark that uses the handwritten digits and stresses several important network capabilities.