A Variational Approach to Mutual Information-Based Coordination for Multi-Agent Reinforcement Learning
This addresses coordination challenges in multi-agent systems, offering a novel method for improving performance in tasks requiring high-quality coordination, though it is incremental as it builds on existing mutual information and variational approaches.
The paper tackles the problem of enabling coordinated behaviors in multi-agent reinforcement learning by proposing a mutual information framework that regularizes returns with mutual information between actions, resulting in VM3-AC outperforming other MARL algorithms in coordination tasks.
In this paper, we propose a new mutual information framework for multi-agent reinforcement learning to enable multiple agents to learn coordinated behaviors by regularizing the accumulated return with the simultaneous mutual information between multi-agent actions. By introducing a latent variable to induce nonzero mutual information between multi-agent actions and applying a variational bound, we derive a tractable lower bound on the considered MMI-regularized objective function. The derived tractable objective can be interpreted as maximum entropy reinforcement learning combined with uncertainty reduction of other agents actions. Applying policy iteration to maximize the derived lower bound, we propose a practical algorithm named variational maximum mutual information multi-agent actor-critic, which follows centralized learning with decentralized execution. We evaluated VM3-AC for several games requiring coordination, and numerical results show that VM3-AC outperforms other MARL algorithms in multi-agent tasks requiring high-quality coordination.