Global Optimality of Elman-type RNN in the Mean-Field Regime
This provides theoretical guarantees for training wide RNNs, which is incremental as it extends mean-field analysis to RNNs.
The paper proves that gradient descent training of wide Elman-type RNNs in the mean-field regime converges to globally optimal fixed points under certain initialization assumptions, establishing optimality for feature-learning.
We analyze Elman-type Recurrent Reural Networks (RNNs) and their training in the mean-field regime. Specifically, we show convergence of gradient descent training dynamics of the RNN to the corresponding mean-field formulation in the large width limit. We also show that the fixed points of the limiting infinite-width dynamics are globally optimal, under some assumptions on the initialization of the weights. Our results establish optimality for feature-learning with wide RNNs in the mean-field regime