Manipulate-Anything: Automating Real-World Robots using Vision-Language Models
This addresses the challenge of scalable data generation for robotics, enabling more efficient training of robots for diverse manipulation tasks, though it is incremental as it builds on prior vision-language model approaches.
The paper tackles the problem of generating high-quality robot demonstration data by proposing Manipulate-Anything, a method that automates trajectory generation for real-world robotic manipulation without requiring privileged state information or hand-designed skills, achieving success on all 7 real-world and 14 simulation tasks and training more robust policies than human demonstrations or existing methods.
Large-scale endeavors like and widespread community efforts such as Open-X-Embodiment have contributed to growing the scale of robot demonstration data. However, there is still an opportunity to improve the quality, quantity, and diversity of robot demonstration data. Although vision-language models have been shown to automatically generate demonstration data, their utility has been limited to environments with privileged state information, they require hand-designed skills, and are limited to interactions with few object instances. We propose Manipulate-Anything, a scalable automated generation method for real-world robotic manipulation. Unlike prior work, our method can operate in real-world environments without any privileged state information, hand-designed skills, and can manipulate any static object. We evaluate our method using two setups. First, Manipulate-Anything successfully generates trajectories for all 7 real-world and 14 simulation tasks, significantly outperforming existing methods like VoxPoser. Second, Manipulate-Anything's demonstrations can train more robust behavior cloning policies than training with human demonstrations, or from data generated by VoxPoser, Scaling-up, and Code-As-Policies. We believe Manipulate-Anything can be a scalable method for both generating data for robotics and solving novel tasks in a zero-shot setting. Project page: https://robot-ma.github.io/.