Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methods
This provides a theoretical foundation for scalable multi-agent control, though it is incremental as it focuses on a specific linear-quadratic case.
The paper tackles the problem of reinforcement learning for many exchangeable agents interacting in a mean-field manner, such as controlling robots via a central unit, by proving the convergence of policy gradient methods in a linear-quadratic setting and establishing convergence rate bounds.
We investigate reinforcement learning in the setting of Markov decision processes for a large number of exchangeable agents interacting in a mean field manner. Applications include, for example, the control of a large number of robots communicating through a central unit dispatching the optimal policy computed by maximizing an aggregate reward. An approximate solution is obtained by learning the optimal policy of a generic agent interacting with the statistical distribution of the states and actions of the other agents. We first provide a full analysis this discrete-time mean field control problem. We then rigorously prove the convergence of exact and model-free policy gradient methods in a mean-field linear-quadratic setting and establish bounds on the rates of convergence. We also provide graphical evidence of the convergence based on implementations of our algorithms.