Contracting Implicit Recurrent Neural Networks: Stable Models with Improved Trainability
This work addresses stability issues in RNNs, which is crucial for trainability and safety in applications like control systems, but it is incremental as it builds on existing contraction analysis methods.
The authors tackled the trade-off between stability and expressibility in recurrent neural networks by proposing an implicit model structure with a convex parametrization for stable models, resulting in significant improvements in training speed and performance compared to previous stable and vanilla RNNs.
Stability of recurrent models is closely linked with trainability, generalizability and in some applications, safety. Methods that train stable recurrent neural networks, however, do so at a significant cost to expressibility. We propose an implicit model structure that allows for a convex parametrization of stable models using contraction analysis of non-linear systems. Using these stability conditions we propose a new approach to model initialization and then provide a number of empirical results comparing the performance of our proposed model set to previous stable RNNs and vanilla RNNs. By carefully controlling stability in the model, we observe a significant increase in the speed of training and model performance.