Learning Mean-Field Games through Mean-Field Actor-Critic Flow
This work addresses computing equilibria in mean-field games, which is important for modeling large-scale multi-agent systems, but appears incremental as it builds on existing techniques like actor-critic methods and optimal transport.
The paper tackles solving mean-field games by proposing the Mean-Field Actor-Critic flow, a continuous-time learning dynamics that combines reinforcement learning and optimal transport, and demonstrates global exponential convergence under theoretical analysis and effectiveness in numerical experiments.
We propose the Mean-Field Actor-Critic (MFAC) flow, a continuous-time learning dynamics for solving mean-field games (MFGs), combining techniques from reinforcement learning and optimal transport. The MFAC framework jointly evolves the control (actor), value function (critic), and distribution components through coupled gradient-based updates governed by partial differential equations (PDEs). A central innovation is the Optimal Transport Geodesic Picard (OTGP) flow, which drives the distribution toward equilibrium along Wasserstein-2 geodesics. We conduct a rigorous convergence analysis using Lyapunov functionals and establish global exponential convergence of the MFAC flow under a suitable timescale. Our results highlight the algorithmic interplay among actor, critic, and distribution components. Numerical experiments illustrate the theoretical findings and demonstrate the effectiveness of the MFAC framework in computing MFG equilibria.