Centralized Permutation Equivariant Policy for Cooperative Multi-Agent Reinforcement Learning
This addresses performance and scalability problems for researchers and practitioners in cooperative multi-agent systems, offering an incremental improvement over existing CTDE methods.
The paper tackles the suboptimal performance and scalability issues in multi-agent reinforcement learning by proposing a centralized training and execution framework with a permutation equivariant architecture, which improves standard algorithms across benchmarks like MPE, SMAC, and RWARE, matching state-of-the-art performance in RWARE.
The Centralized Training with Decentralized Execution (CTDE) paradigm has gained significant attention in multi-agent reinforcement learning (MARL) and is the foundation of many recent algorithms. However, decentralized policies operate under partial observability and often yield suboptimal performance compared to centralized policies, while fully centralized approaches typically face scalability challenges as the number of agents increases. We propose Centralized Permutation Equivariant (CPE) learning, a centralized training and execution framework that employs a fully centralized policy to overcome these limitations. Our approach leverages a novel permutation equivariant architecture, Global-Local Permutation Equivariant (GLPE) networks, that is lightweight, scalable, and easy to implement. Experiments show that CPE integrates seamlessly with both value decomposition and actor-critic methods, substantially improving the performance of standard CTDE algorithms across cooperative benchmarks including MPE, SMAC, and RWARE, and matching the performance of state-of-the-art RWARE implementations.