SAM3D: Segment Anything in 3D Scenes
This work addresses 3D scene segmentation for computer vision applications, but it is incremental as it adapts an existing 2D model to 3D without new training.
The paper tackles the problem of segmenting 3D point clouds by leveraging the Segment-Anything Model (SAM) from 2D RGB images without additional training, achieving reasonable and fine-grained 3D segmentation results as demonstrated on the ScanNet dataset.
In this work, we propose SAM3D, a novel framework that is able to predict masks in 3D point clouds by leveraging the Segment-Anything Model (SAM) in RGB images without further training or finetuning. For a point cloud of a 3D scene with posed RGB images, we first predict segmentation masks of RGB images with SAM, and then project the 2D masks into the 3D points. Later, we merge the 3D masks iteratively with a bottom-up merging approach. At each step, we merge the point cloud masks of two adjacent frames with the bidirectional merging approach. In this way, the 3D masks predicted from different frames are gradually merged into the 3D masks of the whole 3D scene. Finally, we can optionally ensemble the result from our SAM3D with the over-segmentation results based on the geometric information of the 3D scenes. Our approach is experimented with ScanNet dataset and qualitative results demonstrate that our SAM3D achieves reasonable and fine-grained 3D segmentation results without any training or finetuning of SAM.