RNN-based Online Learning: An Efficient First-Order Optimization Algorithm with a Convergence Guarantee
This work addresses the challenge of training RNNs in online learning environments, offering a theoretically sound solution for applications requiring real-time adaptation.
The paper tackles the problem of online nonlinear regression with recurrent neural networks by introducing an efficient first-order training algorithm that guarantees convergence to optimal parameters, achieving significant performance improvements over state-of-the-art methods in simulations.
We investigate online nonlinear regression with continually running recurrent neural network networks (RNNs), i.e., RNN-based online learning. For RNN-based online learning, we introduce an efficient first-order training algorithm that theoretically guarantees to converge to the optimum network parameters. Our algorithm is truly online such that it does not make any assumption on the learning environment to guarantee convergence. Through numerical simulations, we verify our theoretical results and illustrate significant performance improvements achieved by our algorithm with respect to the state-of-the-art RNN training methods.