Sim-to-Real Grasp Detection with Global-to-Local RGB-D Adaptation
This work addresses the sim-to-real gap for robotic grasping, which is an incremental improvement in domain adaptation for a specific robotics application.
This paper tackles the sim-to-real domain adaptation problem for RGB-D grasp detection by proposing a global-to-local alignment method with grasp prototype adaptation, achieving superior results on the GraspNet-Planar benchmark and in physical environments.
This paper focuses on the sim-to-real issue of RGB-D grasp detection and formulates it as a domain adaptation problem. In this case, we present a global-to-local method to address hybrid domain gaps in RGB and depth data and insufficient multi-modal feature alignment. First, a self-supervised rotation pre-training strategy is adopted to deliver robust initialization for RGB and depth networks. We then propose a global-to-local alignment pipeline with individual global domain classifiers for scene features of RGB and depth images as well as a local one specifically working for grasp features in the two modalities. In particular, we propose a grasp prototype adaptation module, which aims to facilitate fine-grained local feature alignment by dynamically updating and matching the grasp prototypes from the simulation and real-world scenarios throughout the training process. Due to such designs, the proposed method substantially reduces the domain shift and thus leads to consistent performance improvements. Extensive experiments are conducted on the GraspNet-Planar benchmark and physical environment, and superior results are achieved which demonstrate the effectiveness of our method.