Generalization in Mean Field Games by Learning Master Policies
This addresses the scalability and practical application of MFGs for large multi-agent systems, representing an incremental improvement by extending existing methods to handle diverse distributions.
The paper tackles the limitation of Mean Field Games (MFGs) to single initial distributions by learning 'Master policies' that enable optimal agent behavior against any population distribution, proving these policies provide Nash equilibria and demonstrating generalization in numerical examples.
Mean Field Games (MFGs) can potentially scale multi-agent systems to extremely large populations of agents. Yet, most of the literature assumes a single initial distribution for the agents, which limits the practical applications of MFGs. Machine Learning has the potential to solve a wider diversity of MFG problems thanks to generalizations capacities. We study how to leverage these generalization properties to learn policies enabling a typical agent to behave optimally against any population distribution. In reference to the Master equation in MFGs, we coin the term ``Master policies'' to describe them and we prove that a single Master policy provides a Nash equilibrium, whatever the initial distribution. We propose a method to learn such Master policies. Our approach relies on three ingredients: adding the current population distribution as part of the observation, approximating Master policies with neural networks, and training via Reinforcement Learning and Fictitious Play. We illustrate on numerical examples not only the efficiency of the learned Master policy but also its generalization capabilities beyond the distributions used for training.