Quantifying the effects of environment and population diversity in multi-agent reinforcement learning
This work addresses generalization challenges for researchers and practitioners in multi-agent systems, though it is incremental as it builds on existing diversity methods.
The paper investigates how diversity in training environments and co-players affects generalization in multi-agent reinforcement learning, finding that procedural generation improves performance on new levels but can reduce it on training levels, and that increasing population diversity through size or intrinsic motivation sometimes enhances agent performance.
Generalization is a major challenge for multi-agent reinforcement learning. How well does an agent perform when placed in novel environments and in interactions with new co-players? In this paper, we investigate and quantify the relationship between generalization and diversity in the multi-agent domain. Across the range of multi-agent environments considered here, procedurally generating training levels significantly improves agent performance on held-out levels. However, agent performance on the specific levels used in training sometimes declines as a result. To better understand the effects of co-player variation, our experiments introduce a new environment-agnostic measure of behavioral diversity. Results demonstrate that population size and intrinsic motivation are both effective methods of generating greater population diversity. In turn, training with a diverse set of co-players strengthens agent performance in some (but not all) cases.