AgentMixer: Multi-Agent Correlated Policy Factorization
This addresses coordination challenges in multi-agent systems, offering a novel approach to improve collaborative decision-making, though it is incremental as it builds on existing CTDE frameworks.
The paper tackles the problem of coordination in multi-agent reinforcement learning by introducing AgentMixer, a method that enables agents to correlate their policies to avoid asymmetric learning failure, achieving performance that matches or outperforms state-of-the-art methods in benchmarks like Multi-Agent MuJoCo and SMAC-v2.
In multi-agent reinforcement learning, centralized training with decentralized execution (CTDE) methods typically assume that agents make decisions based on their local observations independently, which may not lead to a correlated joint policy with coordination. Coordination can be explicitly encouraged during training and individual policies can be trained to imitate the correlated joint policy. However, this may lead to an \textit{asymmetric learning failure} due to the observation mismatch between the joint and individual policies. Inspired by the concept of correlated equilibrium, we introduce a \textit{strategy modification} called AgentMixer that allows agents to correlate their policies. AgentMixer combines individual partially observable policies into a joint fully observable policy non-linearly. To enable decentralized execution, we introduce \textit{Individual-Global-Consistency} to guarantee mode consistency during joint training of the centralized and decentralized policies and prove that AgentMixer converges to an $ε$-approximate Correlated Equilibrium. In the Multi-Agent MuJoCo, SMAC-v2, Matrix Game, and Predator-Prey benchmarks, AgentMixer outperforms or matches state-of-the-art methods.