3D Point Cloud Pre-training with Knowledge Distillation from 2D Images
This addresses the data scarcity issue in 3D point cloud learning for computer vision applications, representing an incremental improvement by adapting existing 2D techniques to 3D.
The paper tackles the problem of limited pre-training data for 3D point clouds by proposing a knowledge distillation method that transfers knowledge from 2D image models like CLIP to 3D models, achieving higher accuracy than state-of-the-art 3D pre-training methods on downstream tasks such as object classification and segmentation.
The recent success of pre-trained 2D vision models is mostly attributable to learning from large-scale datasets. However, compared with 2D image datasets, the current pre-training data of 3D point cloud is limited. To overcome this limitation, we propose a knowledge distillation method for 3D point cloud pre-trained models to acquire knowledge directly from the 2D representation learning model, particularly the image encoder of CLIP, through concept alignment. Specifically, we introduce a cross-attention mechanism to extract concept features from 3D point cloud and compare them with the semantic information from 2D images. In this scheme, the point cloud pre-trained models learn directly from rich information contained in 2D teacher models. Extensive experiments demonstrate that the proposed knowledge distillation scheme achieves higher accuracy than the state-of-the-art 3D pre-training methods for synthetic and real-world datasets on downstream tasks, including object classification, object detection, semantic segmentation, and part segmentation.