Using Fractal Neural Networks to Play SimCity 1 and Conway's Game of Life at Variable Scales
This work addresses the challenge of scale generalization in reinforcement learning for simulation games, which is an incremental improvement in domain-specific applications.
The researchers tackled the problem of training reinforcement learning agents to play SimCity 1 and Conway's Game of Life at variable scales, using fractal neural networks to handle different gameboard sizes, and found that the agents could generalize to larger maps than seen in training, though specific performance numbers were not provided.
We introduce gym-city, a Reinforcement Learning environment that uses SimCity 1's game engine to simulate an urban environment, wherein agents might seek to optimize one or a combination of any number of city-wide metrics, on gameboards of various sizes. We focus on population, and analyze our agents' ability to generalize to larger map-sizes than those seen during training. The environment is interactive, allowing a human player to build alongside agents during training and inference, potentially influencing the course of their learning, or manually probing and evaluating their performance. To test our agents' ability to capture distance-agnostic relationships between elements of the gameboard, we design a minigame within the environment which is, by design, unsolvable at large enough scales given strictly local strategies. Given the game engine's extensive use of Cellular Automata, we also train our agents to "play" Conway's Game of Life -- again optimizing for population -- and examine their behaviour at multiple scales. To make our models compatible with variable-scale gameplay, we use Neural Networks with recursive weights and structure -- fractals to be truncated at different depths, dependent upon the size of the gameboard.