Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning
This addresses the problem of exponential complexity in multi-agent games for researchers and practitioners in MARL, offering a novel method to scale learning effectively.
The paper tackles the challenge of scaling multi-agent reinforcement learning (MARL) to large populations by introducing Evolutionary Population Curriculum (EPC), a curriculum learning paradigm that progressively increases the number of training agents and uses an evolutionary approach to improve adaptability, resulting in consistent large-margin outperformance over baselines as agent numbers grow exponentially.
In multi-agent games, the complexity of the environment can grow exponentially as the number of agents increases, so it is particularly challenging to learn good policies when the agent population is large. In this paper, we introduce Evolutionary Population Curriculum (EPC), a curriculum learning paradigm that scales up Multi-Agent Reinforcement Learning (MARL) by progressively increasing the population of training agents in a stage-wise manner. Furthermore, EPC uses an evolutionary approach to fix an objective misalignment issue throughout the curriculum: agents successfully trained in an early stage with a small population are not necessarily the best candidates for adapting to later stages with scaled populations. Concretely, EPC maintains multiple sets of agents in each stage, performs mix-and-match and fine-tuning over these sets and promotes the sets of agents with the best adaptability to the next stage. We implement EPC on a popular MARL algorithm, MADDPG, and empirically show that our approach consistently outperforms baselines by a large margin as the number of agents grows exponentially.