Unified Reinforcement Q-Learning for Mean Field Game and Control Problems
This addresses the challenge of solving complex multi-agent systems in mean field settings for researchers in control theory and reinforcement learning, though it is incremental as it builds on existing MFG/MFC frameworks.
The authors tackled infinite horizon asymptotic Mean Field Game (MFG) and Mean Field Control (MFC) problems by developing a unified Reinforcement Learning algorithm that learns solutions for both by adjusting learning parameters, achieving results benchmarked against explicit linear-quadratic solutions.
We present a Reinforcement Learning (RL) algorithm to solve infinite horizon asymptotic Mean Field Game (MFG) and Mean Field Control (MFC) problems. Our approach can be described as a unified two-timescale Mean Field Q-learning: The \emph{same} algorithm can learn either the MFG or the MFC solution by simply tuning the ratio of two learning parameters. The algorithm is in discrete time and space where the agent not only provides an action to the environment but also a distribution of the state in order to take into account the mean field feature of the problem. Importantly, we assume that the agent can not observe the population's distribution and needs to estimate it in a model-free manner. The asymptotic MFG and MFC problems are also presented in continuous time and space, and compared with classical (non-asymptotic or stationary) MFG and MFC problems. They lead to explicit solutions in the linear-quadratic (LQ) case that are used as benchmarks for the results of our algorithm.