The Cell Must Go On: Agar.io for Continual Reinforcement Learning
This provides a new, complex simulator for researchers in continual RL, addressing a gap in available tools, though it is incremental as it adapts an existing game rather than proposing a novel method.
The paper tackles the lack of suitable simulators for continual reinforcement learning by introducing AgarCL, a research platform based on Agar.io, which features non-episodic, high-dimensional, and stochastic dynamics, and provides benchmark results for DQN, PPO, and SAC on both the main continual RL problem and smaller isolated tasks.
Continual reinforcement learning (RL) concerns agents that are expected to learn continually, rather than converge to a policy that is then fixed for evaluation. Such an approach is well suited to environments the agent perceives as changing, which renders any static policy ineffective over time. The few simulators explicitly designed for empirical research in continual RL are often limited in scope or complexity, and it is now common for researchers to modify episodic RL environments by artificially incorporating abrupt task changes during interaction. In this paper, we introduce AgarCL, a research platform for continual RL that allows for a progression of increasingly sophisticated behaviour. AgarCL is based on the game Agar.io, a non-episodic, high-dimensional problem featuring stochastic, ever-evolving dynamics, continuous actions, and partial observability. Additionally, we provide benchmark results reporting the performance of DQN, PPO, and SAC in both the primary, challenging continual RL problem, and across a suite of smaller tasks within AgarCL, each of which isolates aspects of the full environment and allow us to characterize the challenges posed by different aspects of the game.