Superhuman AI for Generals.io Using Self-Play Reinforcement Learning
This work demonstrates that with a sufficiently fast simulator, self-play reinforcement learning can achieve superhuman performance in a complex imperfect-information game, providing insights into what matters for such systems.
The authors built a superhuman AI for the real-time strategy game Generals.io, reaching #1 on the public 1v1 leaderboard and beating top human players with a 199-70 record. The key enabler was a JAX-native simulator achieving a 10,000x speedup over prior work.
We present a superhuman AI agent for Generals.io, a real-time strategy game that requires both long-horizon planning and short-term tactics under strong imperfect information. Trained for four days on 4x NVIDIA H200 GPUs, our agent reaches #1 on the public 1v1 leaderboard of over 5,000 human players, leading the second-ranked player by the same margin that separates second place from 25th, and beats the two top-ranked humans head-to-head with a combined 199-70 record across 269 ladder matches. A key enabler is a JAX-native simulator that reaches tens of millions of frames per second on a single GPU, roughly a 10,000x speedup over the prior simulator. On top of this, we train a vision transformer policy end-to-end by self-play with a policy-gradient loop and sparse win/loss reward, using top-advantage sample filtering and an exponential moving average of the policy parameters. Taken together, our findings highlight what matters, and what does not, once a fast simulator removes the data bottleneck.