Neural Fictitious Self-Play on ELF Mini-RTS
This is an incremental step toward game-theoretic solutions for RTS games, addressing the multi-agent nature that hinders single-agent methods.
The paper tackled the challenge of applying AI to real-time strategy (RTS) games by using Neural Fictitious Self-Play (NFSP) on Mini-RTS, showing that NFSP combined with policy gradient reinforcement learning can be effectively applied, with pretraining improving scalability.
Despite the notable successes in video games such as Atari 2600, current AI is yet to defeat human champions in the domain of real-time strategy (RTS) games. One of the reasons is that an RTS game is a multi-agent game, in which single-agent reinforcement learning methods cannot simply be applied because the environment is not a stationary Markov Decision Process. In this paper, we present a first step toward finding a game-theoretic solution to RTS games by applying Neural Fictitious Self-Play (NFSP), a game-theoretic approach for finding Nash equilibria, to Mini-RTS, a small but nontrivial RTS game provided on the ELF platform. More specifically, we show that NFSP can be effectively combined with policy gradient reinforcement learning and be applied to Mini-RTS. Experimental results also show that the scalability of NFSP can be substantially improved by pretraining the models with simple self-play using policy gradients, which by itself gives a strong strategy despite its lack of theoretical guarantee of convergence.