Stochastic Gradient Descent Learns State Equations with Nonlinear Activations
This provides a theoretical foundation for training recurrent neural networks, addressing a core problem in sequential learning tasks, though it is incremental as it builds on existing SGD analysis with new insights for nonlinear activations.
The paper tackles the problem of learning weight matrices in discrete-time dynamical systems, which are fundamental to recurrent neural networks like LSTMs, by proving that stochastic gradient descent (SGD) linearly converges to the ground truth weights with near-optimal sample size for increasing activations with bounded derivatives, as verified numerically for ReLU and leaky ReLU.
We study discrete time dynamical systems governed by the state equation $h_{t+1}=φ(Ah_t+Bu_t)$. Here $A,B$ are weight matrices, $φ$ is an activation function, and $u_t$ is the input data. This relation is the backbone of recurrent neural networks (e.g. LSTMs) which have broad applications in sequential learning tasks. We utilize stochastic gradient descent to learn the weight matrices from a finite input/state trajectory $(u_t,h_t)_{t=0}^N$. We prove that SGD estimate linearly converges to the ground truth weights while using near-optimal sample size. Our results apply to increasing activations whose derivatives are bounded away from zero. The analysis is based on i) a novel SGD convergence result with nonlinear activations and ii) careful statistical characterization of the state vector. Numerical experiments verify the fast convergence of SGD on ReLU and leaky ReLU in consistence with our theory.