On the dimension of pullback attractors in recurrent neural networks
This work provides a theoretical foundation for dimensionality reduction in RNNs and estimating fractal dimensions from time series, contributing incrementally to the understanding of reservoir computers.
The paper tackles the problem of understanding the embedding properties of recurrent neural networks (RNNs) by establishing an upper bound for the fractal dimension of the subset of reservoir state space approximated during training and prediction, proving it is bounded above by the input dimension Nin when input sequences come from an invertible dynamical system.
Recurrent Neural Networks (RNNs) are high-dimensional state space models capable of learning functions on sequence data. Recently, it has been conjectured that reservoir computers, a particular class of RNNs, trained on observations of a dynamical systems can be interpreted as embeddings. This result has been established for the case of linear reservoir systems. In this work, we use a nonautonomous dynamical systems approach to establish an upper bound for the fractal dimension of the subset of reservoir state space approximated during training and prediction phase. We prove that when the input sequences comes from an Nin-dimensional invertible dynamical system, the fractal dimension of this set is bounded above by Nin. The result obtained here are useful in dimensionality reduction of computation in RNNs as well as estimating fractal dimensions of dynamical systems from limited observations of their time series. It is also a step towards understanding embedding properties of reservoir computers.