Entropy Regularization for Mean Field Games with Learning
This work addresses stability and convergence issues in multi-agent reinforcement learning for mean field games, offering incremental improvements.
The paper analyzes the impact of entropy regularization on Mean Field Games with learning, showing that it yields time-dependent policies and stabilizes and accelerates convergence to equilibrium, with a policy-gradient algorithm enabling optimal exploration scheduling.
Entropy regularization has been extensively adopted to improve the efficiency, the stability, and the convergence of algorithms in reinforcement learning. This paper analyzes both quantitatively and qualitatively the impact of entropy regularization for Mean Field Game (MFG) with learning in a finite time horizon. Our study provides a theoretical justification that entropy regularization yields time-dependent policies and, furthermore, helps stabilizing and accelerating convergence to the game equilibrium. In addition, this study leads to a policy-gradient algorithm for exploration in MFG. Under this algorithm, agents are able to learn the optimal exploration scheduling, with stable and fast convergence to the game equilibrium.