Global Convergence of Policy Gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Time
This work addresses the challenge of scaling reinforcement learning to many agents by providing theoretical guarantees for continuous-time models, though it is incremental as it builds on existing discrete-time approaches.
The paper tackles the problem of learning optimal policies in large-scale multi-agent systems by analyzing policy gradient methods for linear-quadratic mean-field control and game models in continuous time, showing convergence to optimal solutions at a linear rate as verified by synthetic simulations.
Reinforcement learning is a powerful tool to learn the optimal policy of possibly multiple agents by interacting with the environment. As the number of agents grow to be very large, the system can be approximated by a mean-field problem. Therefore, it has motivated new research directions for mean-field control (MFC) and mean-field game (MFG). In this paper, we study the policy gradient method for the linear-quadratic mean-field control and game, where we assume each agent has identical linear state transitions and quadratic cost functions. While most of the recent works on policy gradient for MFC and MFG are based on discrete-time models, we focus on the continuous-time models where some analyzing techniques can be interesting to the readers. For both MFC and MFG, we provide policy gradient update and show that it converges to the optimal solution at a linear rate, which is verified by a synthetic simulation. For MFG, we also provide sufficient conditions for the existence and uniqueness of the Nash equilibrium.