Point Clouds Are Specialized Images: A Knowledge Transfer Approach for 3D Understanding
This work addresses 3D data scarcity and high annotation costs for point cloud understanding, offering a novel transfer approach that is incremental in leveraging existing image models.
The paper tackles the problem of 3D point cloud understanding by reinterpreting point clouds as specialized images, enabling knowledge transfer from image data to reduce annotation costs and data scarcity, resulting in state-of-the-art performance with fewer parameters, such as 90.02% accuracy on ScanObjectNN under linear fine-tuning.
Self-supervised representation learning (SSRL) has gained increasing attention in point cloud understanding, in addressing the challenges posed by 3D data scarcity and high annotation costs. This paper presents PCExpert, a novel SSRL approach that reinterprets point clouds as "specialized images". This conceptual shift allows PCExpert to leverage knowledge derived from large-scale image modality in a more direct and deeper manner, via extensively sharing the parameters with a pre-trained image encoder in a multi-way Transformer architecture. The parameter sharing strategy, combined with a novel pretext task for pre-training, i.e., transformation estimation, empowers PCExpert to outperform the state of the arts in a variety of tasks, with a remarkable reduction in the number of trainable parameters. Notably, PCExpert's performance under LINEAR fine-tuning (e.g., yielding a 90.02% overall accuracy on ScanObjectNN) has already approached the results obtained with FULL model fine-tuning (92.66%), demonstrating its effective and robust representation capability.