Deep Policy Iteration for High-Dimensional Mean-Field Games with Regenerative Reformulation
For researchers in mean-field games and multi-agent reinforcement learning, this method provides a scalable approach to high-dimensional problems, avoiding traditional computational bottlenecks.
This paper proposes a deep policy iteration method for high-dimensional finite-horizon mean-field games, reformulating the game as a regenerative problem to enable cycle-by-cycle policy evaluation, improvement, and population measure estimation. The method handles dimensions up to 10,000 in numerical experiments.
This paper develops a deep policy iteration method for high-dimensional finite-horizon mean-field games. We reformulate the game as a regenerative problem with deterministic cycles, which allows policy evaluation (PE), policy improvement (PI), and population measure estimation to be carried out cycle by cycle. Within this formulation, we approximate the population measure by a particle system and update it using a one-step random mapping induced by the Euler-Maruyama discretization of the state dynamics. This update transports a mini-batch of particles from one cycle to the next, avoiding sequential trajectory simulation over the entire time horizon at each iteration. The PE and PI subproblems are formulated through the relation between consecutive cycles, with adversarial training used for evaluation and averaged optimization used for improvement. The resulting method is efficient and scalable in high dimensions, as it avoids the direct solution of the coupled Hamilton-Jacobi-Bellman and Fokker-Planck system, the full simulation of trajectories to estimate the population measure, the explicit computation of conditional expectations in policy evaluation, and pointwise optimization in policy improvement. Numerical experiments demonstrate that the proposed method effectively handles dimensions up to 10,000.