SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Instance Segmentation
This addresses the problem of segmenting 3D instances without training for applications in 3D scene understanding, though it is incremental as it builds on SAM.
The paper tackles zero-shot instance segmentation of 3D scenes by applying the Segment Anything Model (SAM) to 2D frames and aligning prompts in 3D for view consistency, achieving comparable or better performance than previous zero-shot or supervised methods and surpassing human annotations in many cases.
We introduce SAMPro3D for zero-shot instance segmentation of 3D scenes. Given the 3D point cloud and multiple posed RGB-D frames of 3D scenes, our approach segments 3D instances by applying the pretrained Segment Anything Model (SAM) to 2D frames. Our key idea involves locating SAM prompts in 3D to align their projected pixel prompts across frames, ensuring the view consistency of SAM-predicted masks. Moreover, we suggest selecting prompts from the initial set guided by the information of SAM-predicted masks across all views, which enhances the overall performance. We further propose to consolidate different prompts if they are segmenting different surface parts of the same 3D instance, bringing a more comprehensive segmentation. Notably, our method does not require any additional training. Extensive experiments on diverse benchmarks show that our method achieves comparable or better performance compared to previous zero-shot or fully supervised approaches, and in many cases surpasses human annotations. Furthermore, since our fine-grained predictions often lack annotations in available datasets, we present ScanNet200-Fine50 test data which provides fine-grained annotations on 50 scenes from ScanNet200 dataset. The project page can be accessed at https://mutianxu.github.io/sampro3d/.