TLeague: A Framework for Competitive Self-Play based Distributed Multi-Agent Reinforcement Learning
This framework addresses the computational bottleneck of data-thirsty competitive self-play MARL for researchers and engineers, making it more accessible for real-world applications.
This paper introduces TLeague, a distributed multi-agent reinforcement learning framework designed to accelerate competitive self-play (CSP) training, which typically requires billions of environmental frames. TLeague supports large-scale training on single machines or hybrid clusters, achieving high throughput and reasonable scale-up, as demonstrated through experiments on StarCraft II, ViZDoom, and Pommerman.
Competitive Self-Play (CSP) based Multi-Agent Reinforcement Learning (MARL) has shown phenomenal breakthroughs recently. Strong AIs are achieved for several benchmarks, including Dota 2, Glory of Kings, Quake III, StarCraft II, to name a few. Despite the success, the MARL training is extremely data thirsty, requiring typically billions of (if not trillions of) frames be seen from the environment during training in order for learning a high performance agent. This poses non-trivial difficulties for researchers or engineers and prevents the application of MARL to a broader range of real-world problems. To address this issue, in this manuscript we describe a framework, referred to as TLeague, that aims at large-scale training and implements several main-stream CSP-MARL algorithms. The training can be deployed in either a single machine or a cluster of hybrid machines (CPUs and GPUs), where the standard Kubernetes is supported in a cloud native manner. TLeague achieves a high throughput and a reasonable scale-up when performing distributed training. Thanks to the modular design, it is also easy to extend for solving other multi-agent problems or implementing and verifying MARL algorithms. We present experiments over StarCraft II, ViZDoom and Pommerman to show the efficiency and effectiveness of TLeague. The code is open-sourced and available at https://github.com/tencent-ailab/tleague_projpage