Stable Recurrent Models
This work addresses the fundamental role of stability in recurrent models for sequence learning, potentially explaining why feed-forward models often succeed in practice.
The authors investigated stable recurrent neural networks, proving they can be approximated by feed-forward networks for inference and training, and empirically showed they perform similarly to unstable models on benchmark tasks.
Stability is a fundamental property of dynamical systems, yet to this date it has had little bearing on the practice of recurrent neural networks. In this work, we conduct a thorough investigation of stable recurrent models. Theoretically, we prove stable recurrent neural networks are well approximated by feed-forward networks for the purpose of both inference and training by gradient descent. Empirically, we demonstrate stable recurrent models often perform as well as their unstable counterparts on benchmark sequence tasks. Taken together, these findings shed light on the effective power of recurrent networks and suggest much of sequence learning happens, or can be made to happen, in the stable regime. Moreover, our results help to explain why in many cases practitioners succeed in replacing recurrent models by feed-forward models.