Major-Minor Mean Field Multi-Agent Reinforcement Learning
This addresses the problem of scaling cooperative MARL to many agents, offering a more flexible approach than strict mean field control, though it appears incremental as it builds upon existing MFC frameworks.
The paper tackles the scalability issue in multi-agent reinforcement learning (MARL) by introducing Major-Minor Mean Field Control (M3FC) to model many similar and few complex agents, and demonstrates strong performance compared to state-of-the-art methods in experiments.
Multi-agent reinforcement learning (MARL) remains difficult to scale to many agents. Recent MARL using Mean Field Control (MFC) provides a tractable and rigorous approach to otherwise difficult cooperative MARL. However, the strict MFC assumption of many independent, weakly-interacting agents is too inflexible in practice. We generalize MFC to instead simultaneously model many similar and few complex agents -- as Major-Minor Mean Field Control (M3FC). Theoretically, we give approximation results for finite agent control, and verify the sufficiency of stationary policies for optimality together with a dynamic programming principle. Algorithmically, we propose Major-Minor Mean Field MARL (M3FMARL) for finite agent systems instead of the limiting system. The algorithm is shown to approximate the policy gradient of the underlying M3FC MDP. Finally, we demonstrate its capabilities experimentally in various scenarios. We observe a strong performance in comparison to state-of-the-art policy gradient MARL methods.