Diversity is Strength: Mastering Football Full Game with Interactive Reinforcement Learning of Multiple AIs
This addresses the problem of strategy diversity in multi-agent reinforcement learning for AI competitions, though it appears incremental as it builds on existing DRL methods with a novel training structure.
The authors tackled the challenge of training AI with diverse and strong strategies in multi-agent environments by proposing the Diversity is Strength (DIS) framework, which won the 5v5 and 11v11 tracks in the Google Research Football AI competition, enabling high performance in both for the first time.
Training AI with strong and rich strategies in multi-agent environments remains an important research topic in Deep Reinforcement Learning (DRL). The AI's strength is closely related to its diversity of strategies, and this relationship can guide us to train AI with both strong and rich strategies. To prove this point, we propose Diversity is Strength (DIS), a novel DRL training framework that can simultaneously train multiple kinds of AIs. These AIs are linked through an interconnected history model pool structure, which enhances their capabilities and strategy diversities. We also design a model evaluation and screening scheme to select the best models to enrich the model pool and obtain the final AI. The proposed training method provides diverse, generalizable, and strong AI strategies without using human data. We tested our method in an AI competition based on Google Research Football (GRF) and won the 5v5 and 11v11 tracks. The method enables a GRF AI to have a high level on both 5v5 and 11v11 tracks for the first time, which are under complex multi-agent environments. The behavior analysis shows that the trained AI has rich strategies, and the ablation experiments proved that the designed modules benefit the training process.