Minimax Exploiter: A Data Efficient Approach for Competitive Self-Play
This work addresses the impracticality of deploying CSP methods in iterative real-life settings like video game productions by enhancing data efficiency, though it is incremental as it builds on existing CSP frameworks.
The paper tackles the high computational cost and time required for training Competitive Self-Play (CSP) methods in multi-agent reinforcement learning by proposing the Minimax Exploiter, a game-theoretic approach that leverages opponent knowledge to improve data efficiency, resulting in consistent outperformance of baselines in various game settings.
Recent advances in Competitive Self-Play (CSP) have achieved, or even surpassed, human level performance in complex game environments such as Dota 2 and StarCraft II using Distributed Multi-Agent Reinforcement Learning (MARL). One core component of these methods relies on creating a pool of learning agents -- consisting of the Main Agent, past versions of this agent, and Exploiter Agents -- where Exploiter Agents learn counter-strategies to the Main Agents. A key drawback of these approaches is the large computational cost and physical time that is required to train the system, making them impractical to deploy in highly iterative real-life settings such as video game productions. In this paper, we propose the Minimax Exploiter, a game theoretic approach to exploiting Main Agents that leverages knowledge of its opponents, leading to significant increases in data efficiency. We validate our approach in a diversity of settings, including simple turn based games, the arcade learning environment, and For Honor, a modern video game. The Minimax Exploiter consistently outperforms strong baselines, demonstrating improved stability and data efficiency, leading to a robust CSP-MARL method that is both flexible and easy to deploy.