Take-A-Photo: 3D-to-2D Generative Pre-training of Point Cloud Models
This addresses the problem of limited generative pre-training in 3D vision for researchers and practitioners, offering a novel approach that is adaptable to any point cloud model, though it builds incrementally on existing 2D methods like MAE.
The paper tackles the challenge of generative pre-training for point cloud models by proposing a 3D-to-2D method that generates view images from instructed poses, which improves performance on tasks like ScanObjectNN classification and ShapeNetPart segmentation, achieving state-of-the-art results.
With the overwhelming trend of mask image modeling led by MAE, generative pre-training has shown a remarkable potential to boost the performance of fundamental models in 2D vision. However, in 3D vision, the over-reliance on Transformer-based backbones and the unordered nature of point clouds have restricted the further development of generative pre-training. In this paper, we propose a novel 3D-to-2D generative pre-training method that is adaptable to any point cloud model. We propose to generate view images from different instructed poses via the cross-attention mechanism as the pre-training scheme. Generating view images has more precise supervision than its point cloud counterpart, thus assisting 3D backbones to have a finer comprehension of the geometrical structure and stereoscopic relations of the point cloud. Experimental results have proved the superiority of our proposed 3D-to-2D generative pre-training over previous pre-training methods. Our method is also effective in boosting the performance of architecture-oriented approaches, achieving state-of-the-art performance when fine-tuning on ScanObjectNN classification and ShapeNetPart segmentation tasks. Code is available at https://github.com/wangzy22/TAP.