CL3R: 3D Reconstruction and Contrastive Learning for Enhanced Robotic Manipulation Representations
This addresses the need for robust perception in fine-grained robotic manipulation, though it appears incremental as it builds on existing 2D foundation models and 3D representation methods.
The paper tackles the problem of robotic manipulation policies struggling with 3D spatial information and viewpoint generalization by proposing CL3R, a 3D pre-training framework that integrates spatial awareness and semantic understanding, resulting in enhanced performance in simulation and real-world experiments.
Building a robust perception module is crucial for visuomotor policy learning. While recent methods incorporate pre-trained 2D foundation models into robotic perception modules to leverage their strong semantic understanding, they struggle to capture 3D spatial information and generalize across diverse camera viewpoints. These limitations hinder the policy's effectiveness, especially in fine-grained robotic manipulation scenarios. To address these challenges, we propose CL3R, a novel 3D pre-training framework designed to enhance robotic manipulation policies. Our method integrates both spatial awareness and semantic understanding by employing a point cloud Masked Autoencoder to learn rich 3D representations while leveraging pre-trained 2D foundation models through contrastive learning for efficient semantic knowledge transfer. Additionally, we propose a 3D visual representation pre-training framework for robotic tasks. By unifying coordinate systems across datasets and introducing random fusion of multi-view point clouds, we mitigate camera view ambiguity and improve generalization, enabling robust perception from novel viewpoints at test time. Extensive experiments in both simulation and the real world demonstrate the superiority of our method, highlighting its effectiveness in visuomotor policy learning for robotic manipulation.