NeuPL: Neural Population Learning
This work solves the problem of inefficient policy learning in complex strategy games for AI researchers and practitioners, though it appears incremental as it builds on existing iterative approaches.
The paper tackles the problem of discovering diverse policies in strategy games like StarCraft and poker by addressing inefficiencies in iterative training methods, proposing Neural Population Learning (NeuPL) which offers convergence guarantees and enables transfer learning, resulting in improved performance and efficiency across test domains.
Learning in strategy games (e.g. StarCraft, poker) requires the discovery of diverse policies. This is often achieved by iteratively training new policies against existing ones, growing a policy population that is robust to exploit. This iterative approach suffers from two issues in real-world games: a) under finite budget, approximate best-response operators at each iteration needs truncating, resulting in under-trained good-responses populating the population; b) repeated learning of basic skills at each iteration is wasteful and becomes intractable in the presence of increasingly strong opponents. In this work, we propose Neural Population Learning (NeuPL) as a solution to both issues. NeuPL offers convergence guarantees to a population of best-responses under mild assumptions. By representing a population of policies within a single conditional model, NeuPL enables transfer learning across policies. Empirically, we show the generality, improved performance and efficiency of NeuPL across several test domains. Most interestingly, we show that novel strategies become more accessible, not less, as the neural population expands.