Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization
This addresses the issue of limited strategy diversity in multi-agent reinforcement learning for games, enabling better performance and adaptability, though it appears incremental as it builds on policy gradient methods.
The paper tackles the problem of discovering diverse strategic policies in complex multi-agent games, where standard algorithms converge to a single sub-optimal equilibrium, by proposing Reward Randomization and Reward-Randomized Policy Gradient (RPG), which successfully finds multiple human-interpretable strategies in grid-world games and Agar.io, leading to higher payoffs and adaptive agents.
We propose a simple, general and effective technique, Reward Randomization for discovering diverse strategic policies in complex multi-agent games. Combining reward randomization and policy gradient, we derive a new algorithm, Reward-Randomized Policy Gradient (RPG). RPG is able to discover multiple distinctive human-interpretable strategies in challenging temporal trust dilemmas, including grid-world games and a real-world game Agar.io, where multiple equilibria exist but standard multi-agent policy gradient algorithms always converge to a fixed one with a sub-optimal payoff for every player even using state-of-the-art exploration techniques. Furthermore, with the set of diverse strategies from RPG, we can (1) achieve higher payoffs by fine-tuning the best policy from the set; and (2) obtain an adaptive agent by using this set of strategies as its training opponents. The source code and example videos can be found in our website: https://sites.google.com/view/staghuntrpg.