$π_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
This addresses the challenge of real-world robotic generalization for practical applications, representing a significant step beyond incremental improvements.
The paper tackles the problem of enabling robots to perform practical tasks in the real world by developing $\pi_{0.5}$, a vision-language-action model that uses co-training on heterogeneous tasks to achieve broad generalization, demonstrating for the first time that an end-to-end learning system can perform long-horizon and dexterous manipulation skills like cleaning in entirely new homes.
In order for robots to be useful, they must perform practically relevant tasks in the real world, outside of the lab. While vision-language-action (VLA) models have demonstrated impressive results for end-to-end robot control, it remains an open question how far such models can generalize in the wild. We describe $π_{0.5}$, a new model based on $π_{0}$ that uses co-training on heterogeneous tasks to enable broad generalization. $π_{0.5}$\ uses data from multiple robots, high-level semantic prediction, web data, and other sources to enable broadly generalizable real-world robotic manipulation. Our system uses a combination of co-training and hybrid multi-modal examples that combine image observations, language commands, object detections, semantic subtask prediction, and low-level actions. Our experiments show that this kind of knowledge transfer is essential for effective generalization, and we demonstrate for the first time that an end-to-end learning-enabled robotic system can perform long-horizon and dexterous manipulation skills, such as cleaning a kitchen or bedroom, in entirely new homes.