Self-Play PSRO: Toward Optimal Populations in Two-Player Zero-Sum Games
This addresses a bottleneck in competitive multi-agent reinforcement learning for researchers and practitioners, offering an incremental improvement over prior methods.
The paper tackled the problem of slow convergence in population-based reinforcement learning methods for two-player zero-sum games by introducing Self-Play PSRO, which adds approximately optimal stochastic policies to the population each iteration, resulting in faster convergence than existing methods like APSRO, often in just a few iterations.
In competitive two-agent environments, deep reinforcement learning (RL) methods based on the \emph{Double Oracle (DO)} algorithm, such as \emph{Policy Space Response Oracles (PSRO)} and \emph{Anytime PSRO (APSRO)}, iteratively add RL best response policies to a population. Eventually, an optimal mixture of these population policies will approximate a Nash equilibrium. However, these methods might need to add all deterministic policies before converging. In this work, we introduce \emph{Self-Play PSRO (SP-PSRO)}, a method that adds an approximately optimal stochastic policy to the population in each iteration. Instead of adding only deterministic best responses to the opponent's least exploitable population mixture, SP-PSRO also learns an approximately optimal stochastic policy and adds it to the population as well. As a result, SP-PSRO empirically tends to converge much faster than APSRO and in many games converges in just a few iterations.