Stackelberg Games for Learning Emergent Behaviors During Competitive Autocurricula
This addresses the problem of low-quality equilibria in autocurricular training for robotics and multi-agent systems, offering a novel approach to enhance emergent behaviors, though it appears incremental as it builds on existing MARL methods.
The paper tackles the challenge of generating sophisticated policies in asymmetric multi-agent reinforcement learning tasks by proposing ST-MADDPG, a game-theoretic algorithm that formulates interactions as a Stackelberg game, resulting in improved co-evolution and strategies effective against unseen opponents.
Autocurricular training is an important sub-area of multi-agent reinforcement learning~(MARL) that allows multiple agents to learn emergent skills in an unsupervised co-evolving scheme. The robotics community has experimented autocurricular training with physically grounded problems, such as robust control and interactive manipulation tasks. However, the asymmetric nature of these tasks makes the generation of sophisticated policies challenging. Indeed, the asymmetry in the environment may implicitly or explicitly provide an advantage to a subset of agents which could, in turn, lead to a low-quality equilibrium. This paper proposes a novel game-theoretic algorithm, Stackelberg Multi-Agent Deep Deterministic Policy Gradient (ST-MADDPG), which formulates a two-player MARL problem as a Stackelberg game with one player as the `leader' and the other as the `follower' in a hierarchical interaction structure wherein the leader has an advantage. We first demonstrate that the leader's advantage from ST-MADDPG can be used to alleviate the inherent asymmetry in the environment. By exploiting the leader's advantage, ST-MADDPG improves the quality of a co-evolution process and results in more sophisticated and complex strategies that work well even against an unseen strong opponent.