Asymptotic Stability in Reservoir Computing
This work addresses stability issues in Reservoir Computing, which is important for practitioners in machine learning and neural networks, but it is incremental as it builds on existing theoretical frameworks.
The paper tackles the problem of stability in Reservoir Computing, a class of Recurrent Neural Networks, by using the recurrent kernel limit to quantitatively characterize the frontier between stability and chaos, which aids hyperparameter tuning.
Reservoir Computing is a class of Recurrent Neural Networks with internal weights fixed at random. Stability relates to the sensitivity of the network state to perturbations. It is an important property in Reservoir Computing as it directly impacts performance. In practice, it is desirable to stay in a stable regime, where the effect of perturbations does not explode exponentially, but also close to the chaotic frontier where reservoir dynamics are rich. Open questions remain today regarding input regularization and discontinuous activation functions. In this work, we use the recurrent kernel limit to draw new insights on stability in reservoir computing. This limit corresponds to large reservoir sizes, and it already becomes relevant for reservoirs with a few hundred neurons. We obtain a quantitative characterization of the frontier between stability and chaos, which can greatly benefit hyperparameter tuning. In a broader sense, our results contribute to understanding the complex dynamics of Recurrent Neural Networks.