Decentralized Transformers with Centralized Aggregation are Sample-Efficient Multi-Agent World Models
This addresses scalability and non-stationarity issues in multi-agent systems, offering a novel approach for improving sample efficiency in MARL, though it is incremental as it builds on existing Transformer and world model techniques.
The paper tackles the challenge of building scalable and consistent world models for multi-agent reinforcement learning by proposing a decentralized Transformer-based model with centralized aggregation, which outperforms existing methods on the Starcraft Multi-Agent Challenge in sample efficiency and performance.
Learning a world model for model-free Reinforcement Learning (RL) agents can significantly improve the sample efficiency by learning policies in imagination. However, building a world model for Multi-Agent RL (MARL) can be particularly challenging due to the scalability issue in a centralized architecture arising from a large number of agents, and also the non-stationarity issue in a decentralized architecture stemming from the inter-dependency among agents. To address both challenges, we propose a novel world model for MARL that learns decentralized local dynamics for scalability, combined with a centralized representation aggregation from all agents. We cast the dynamics learning as an auto-regressive sequence modeling problem over discrete tokens by leveraging the expressive Transformer architecture, in order to model complex local dynamics across different agents and provide accurate and consistent long-term imaginations. As the first pioneering Transformer-based world model for multi-agent systems, we introduce a Perceiver Transformer as an effective solution to enable centralized representation aggregation within this context. Results on Starcraft Multi-Agent Challenge (SMAC) show that it outperforms strong model-free approaches and existing model-based methods in both sample efficiency and overall performance.