Asymptotics of Reinforcement Learning with Neural Networks
This provides foundational theoretical insights into the convergence properties of neural networks in RL, which is incremental as it builds on existing asymptotic analysis methods.
The paper tackles the problem of understanding the asymptotic behavior of neural networks in reinforcement learning, proving that a single-layer network trained with Q-learning converges to a random ordinary differential equation whose stationary solution yields the optimal control, with analysis showing convergence to this solution.
We prove that a single-layer neural network trained with the Q-learning algorithm converges in distribution to a random ordinary differential equation as the size of the model and the number of training steps become large. Analysis of the limit differential equation shows that it has a unique stationary solution which is the solution of the Bellman equation, thus giving the optimal control for the problem. In addition, we study the convergence of the limit differential equation to the stationary solution. As a by-product of our analysis, we obtain the limiting behavior of single-layer neural networks when trained on i.i.d. data with stochastic gradient descent under the widely-used Xavier initialization.