Population-Aware Imitation Learning in Mean-field Games with Common Noise
For researchers in multi-agent reinforcement learning and mean-field games, this work addresses the challenge of stochastic population dynamics in imitation learning, which is an incremental extension of existing methods to a new setting.
This paper tackles imitation learning in mean-field games with common noise, where the population distribution evolves stochastically. It shows that population-aware policies are crucial to avoid being misled by randomness, and that minimizing behavioral cloning or adversarial divergence effectively controls exploitability and performance gap.
Mean Field Games (MFGs) provide a powerful framework for modeling the collective behavior of large populations of interacting agents. In this paper, we address the problem of Imitation Learning (IL) in MFGs subject to common noise, where the population distribution evolves stochastically. This stochasticity compels agents to adopt population-aware policies to respond to aggregate shocks. We formulate two distinct learning objectives: recovering a Nash equilibrium and maximizing performance against an expert population. We investigate two imitation proxies: Behavioral Cloning (BC) and Adversarial (ADV) divergence. We then establish finite-sample error bounds showing that minimizing these proxies effectively controls both the policy's exploitability and its performance gap relative to the expert. Furthermore, we propose a numerical framework using generalized Fictitious Play and Deep Learning to compute expert population-aware policies. Through experiments on three environments we demonstrate that standard population-unaware policies fail to capture the equilibrium dynamics. Our results highlight that learning population-aware policies is crucial to avoid being misled by the randomness inherent in common noise.