SUGAR: Pre-training 3D Visual Representations for Robotics
This addresses the need for better 3D perception in robotics to handle occlusions and object localization in cluttered scenes, offering a novel pre-training approach.
The paper tackles the problem of learning generalizable 3D visual representations for robotics by introducing SUGAR, a framework that pre-trains on multi-object 3D point clouds to capture semantic, geometric, and affordance properties, and it outperforms state-of-the-art 2D and 3D representations on tasks like zero-shot 3D object recognition.
Learning generalizable visual representations from Internet data has yielded promising results for robotics. Yet, prevailing approaches focus on pre-training 2D representations, being sub-optimal to deal with occlusions and accurately localize objects in complex 3D scenes. Meanwhile, 3D representation learning has been limited to single-object understanding. To address these limitations, we introduce a novel 3D pre-training framework for robotics named SUGAR that captures semantic, geometric and affordance properties of objects through 3D point clouds. We underscore the importance of cluttered scenes in 3D representation learning, and automatically construct a multi-object dataset benefiting from cost-free supervision in simulation. SUGAR employs a versatile transformer-based model to jointly address five pre-training tasks, namely cross-modal knowledge distillation for semantic learning, masked point modeling to understand geometry structures, grasping pose synthesis for object affordance, 3D instance segmentation and referring expression grounding to analyze cluttered scenes. We evaluate our learned representation on three robotic-related tasks, namely, zero-shot 3D object recognition, referring expression grounding, and language-driven robotic manipulation. Experimental results show that SUGAR's 3D representation outperforms state-of-the-art 2D and 3D representations.