A recurrent neural network without chaos
This addresses the issue of unpredictable behavior in recurrent neural networks for researchers and practitioners in machine learning, though it appears incremental as it builds on existing gated architectures.
The authors tackled the problem of chaotic dynamics in gated RNNs by introducing a simple gated RNN that achieves performance comparable to LSTMs and GRUs on word-level language modeling, with the key result being a proven non-chaotic and predictable dynamical system.
We introduce an exceptionally simple gated recurrent neural network (RNN) that achieves performance comparable to well-known gated architectures, such as LSTMs and GRUs, on the word-level language modeling task. We prove that our model has simple, predicable and non-chaotic dynamics. This stands in stark contrast to more standard gated architectures, whose underlying dynamical systems exhibit chaotic behavior.