SPECTra: Scalable Multi-Agent Reinforcement Learning with Permutation-Free Networks
This work addresses efficiency and adaptability challenges in cooperative MARL for researchers and practitioners, though it is incremental as it builds on prior methods like GNNs and self-attention.
The paper tackles the permutation problem and scalability issues in cooperative multi-agent reinforcement learning by proposing a novel agent network and non-linear mixing network that ensure permutation-equivariance and scalability, achieving superior learning performance on SMACv2 and Google Research Football benchmarks compared to existing methods.
In cooperative multi-agent reinforcement learning (MARL), the permutation problem where the state space grows exponentially with the number of agents reduces sample efficiency. Additionally, many existing architectures struggle with scalability, relying on a fixed structure tied to a specific number of agents, limiting their applicability to environments with a variable number of entities. While approaches such as graph neural networks (GNNs) and self-attention mechanisms have progressed in addressing these challenges, they have significant limitations as dense GNNs and self-attention mechanisms incur high computational costs. To overcome these limitations, we propose a novel agent network and a non-linear mixing network that ensure permutation-equivariance and scalability, allowing them to generalize to environments with various numbers of agents. Our agent network significantly reduces computational complexity, and our scalable hypernetwork enables efficient weight generation for non-linear mixing. Additionally, we introduce curriculum learning to improve training efficiency. Experiments on SMACv2 and Google Research Football (GRF) demonstrate that our approach achieves superior learning performance compared to existing methods. By addressing both permutation-invariance and scalability in MARL, our work provides a more efficient and adaptable framework for cooperative MARL. Our code is available at https://github.com/funny-rl/SPECTra.