Self-supervised Pre-training with Masked Shape Prediction for 3D Scene Understanding
This work addresses the need for effective self-supervised methods in 3D scene understanding, offering a novel approach that enhances downstream tasks, though it is incremental relative to existing masked modeling techniques in 2D and language domains.
The paper tackled the problem of self-supervised pre-training for 3D scene understanding by introducing Masked Shape Prediction (MSP), which uses geometric shape as a target for masked points, resulting in improved feature representations that consistently boost downstream performance on multiple tasks across indoor and outdoor datasets.
Masked signal modeling has greatly advanced self-supervised pre-training for language and 2D images. However, it is still not fully explored in 3D scene understanding. Thus, this paper introduces Masked Shape Prediction (MSP), a new framework to conduct masked signal modeling in 3D scenes. MSP uses the essential 3D semantic cue, i.e., geometric shape, as the prediction target for masked points. The context-enhanced shape target consisting of explicit shape context and implicit deep shape feature is proposed to facilitate exploiting contextual cues in shape prediction. Meanwhile, the pre-training architecture in MSP is carefully designed to alleviate the masked shape leakage from point coordinates. Experiments on multiple 3D understanding tasks on both indoor and outdoor datasets demonstrate the effectiveness of MSP in learning good feature representations to consistently boost downstream performance.