SAS: Segment Any 3D Scene with Integrated 2D Priors
This addresses the limitation of fixed-category 3D models for dynamic scenes, offering a practical solution for robotics and AR/VR applications, though it builds incrementally on existing 2D-to-3D transfer methods.
The paper tackles the problem of recognizing unseen objects in 3D scenes by integrating open-vocabulary capabilities from 2D models into 3D domain, achieving significant performance improvements across multiple datasets like ScanNet v2 and Matterport3D.
The open vocabulary capability of 3D models is increasingly valued, as traditional methods with models trained with fixed categories fail to recognize unseen objects in complex dynamic 3D scenes. In this paper, we propose a simple yet effective approach, SAS, to integrate the open vocabulary capability of multiple 2D models and migrate it to 3D domain. Specifically, we first propose Model Alignment via Text to map different 2D models into the same embedding space using text as a bridge. Then, we propose Annotation-Free Model Capability Construction to explicitly quantify the 2D model's capability of recognizing different categories using diffusion models. Following this, point cloud features from different 2D models are fused with the guide of constructed model capabilities. Finally, the integrated 2D open vocabulary capability is transferred to 3D domain through feature distillation. SAS outperforms previous methods by a large margin across multiple datasets, including ScanNet v2, Matterport3D, and nuScenes, while its generalizability is further validated on downstream tasks, e.g., gaussian segmentation and instance segmentation.