Learning in ImaginationLand: Omnidirectional Policies through 3D Generative Models (OP-Gen)
This addresses the data efficiency challenge in robotics by leveraging generative models to create synthetic training data, though it is incremental as it builds on existing 3D generative techniques.
The paper tackles the problem of reducing the number of demonstrations needed for robot policy learning by using 3D generative models to augment a single real-world demonstration into an imagined dataset, enabling robots to perform tasks from states far from those observed, such as starting from the opposite side of an object.
Recent 3D generative models, which are capable of generating full object shapes from just a few images, now open up new opportunities in robotics. In this work, we show that 3D generative models can be used to augment a dataset from a single real-world demonstration, after which an omnidirectional policy can be learned within this imagined dataset. We found that this enables a robot to perform a task when initialised from states very far from those observed during the demonstration, including starting from the opposite side of the object relative to the real-world demonstration, significantly reducing the number of demonstrations required for policy learning. Through several real-world experiments across tasks such as grasping objects, opening a drawer, and placing trash into a bin, we study these omnidirectional policies by investigating the effect of various design choices on policy behaviour, and we show superior performance to recent baselines which use alternative methods for data augmentation.