Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks
This work addresses the problem of generalization in reinforcement learning for physical control, offering a scalable framework for pre-training agents, though it is incremental in extending self-supervised learning concepts to RL.
The authors tackled the challenge of training general agents for sequential decision problems by procedurally generating tens of millions of 2D physics-based tasks and training a reinforcement learning agent, which achieved strong physical reasoning and zero-shot solving of unseen environments, with fine-tuning outperforming tabula rasa training.
While large models trained with self-supervised learning on offline datasets have shown remarkable capabilities in text and image domains, achieving the same generalisation for agents that act in sequential decision problems remains an open challenge. In this work, we take a step towards this goal by procedurally generating tens of millions of 2D physics-based tasks and using these to train a general reinforcement learning (RL) agent for physical control. To this end, we introduce Kinetix: an open-ended space of physics-based RL environments that can represent tasks ranging from robotic locomotion and grasping to video games and classic RL environments, all within a unified framework. Kinetix makes use of our novel hardware-accelerated physics engine Jax2D that allows us to cheaply simulate billions of environment steps during training. Our trained agent exhibits strong physical reasoning capabilities in 2D space, being able to zero-shot solve unseen human-designed environments. Furthermore, fine-tuning this general agent on tasks of interest shows significantly stronger performance than training an RL agent *tabula rasa*. This includes solving some environments that standard RL training completely fails at. We believe this demonstrates the feasibility of large scale, mixed-quality pre-training for online RL and we hope that Kinetix will serve as a useful framework to investigate this further.