Policy Diversity for Cooperative Agents
This addresses the need for domain experts to have multiple cooperation strategies in multi-agent systems, though it appears incremental as it builds on existing diversity concepts.
The paper tackles the problem of generating diverse cooperative policies in multi-agent reinforcement learning, proposing Moment-Matching Policy Diversity to produce varying team policies, with effectiveness demonstrated on a team-based shooter.
Standard cooperative multi-agent reinforcement learning (MARL) methods aim to find the optimal team cooperative policy to complete a task. However there may exist multiple different ways of cooperating, which usually are very needed by domain experts. Therefore, identifying a set of significantly different policies can alleviate the task complexity for them. Unfortunately, there is a general lack of effective policy diversity approaches specifically designed for the multi-agent domain. In this work, we propose a method called Moment-Matching Policy Diversity to alleviate this problem. This method can generate different team policies to varying degrees by formalizing the difference between team policies as the difference in actions of selected agents in different policies. Theoretically, we show that our method is a simple way to implement a constrained optimization problem that regularizes the difference between two trajectory distributions by using the maximum mean discrepancy. The effectiveness of our approach is demonstrated on a challenging team-based shooter.