How Chaotic Are Recurrent Neural Networks?
This addresses a theoretical concern for researchers and practitioners using RNNs, but it is incremental as it refutes prior beliefs without introducing new methods.
The paper tackles the problem of whether recurrent neural networks (RNNs) exhibit chaotic behavior in practice, finding through empirical analysis that vanilla and LSTM RNNs do not show chaotic behavior in applications like text generation.
Recurrent neural networks (RNNs) are non-linear dynamic systems. Previous work believes that RNN may suffer from the phenomenon of chaos, where the system is sensitive to initial states and unpredictable in the long run. In this paper, however, we perform a systematic empirical analysis, showing that a vanilla or long short term memory (LSTM) RNN does not exhibit chaotic behavior along the training process in real applications such as text generation. Our findings suggest that future work in this direction should address the other side of non-linear dynamics for RNN.