OpenScene: 3D Scene Understanding with Open Vocabularies
This addresses the problem of limited labeled 3D datasets for researchers and practitioners in computer vision, offering a flexible, task-agnostic model for various scene understanding applications.
The paper tackles 3D scene understanding by proposing OpenScene, a zero-shot approach that predicts dense features for 3D points co-embedded with CLIP, enabling open-vocabulary queries without labeled 3D data, achieving state-of-the-art zero-shot 3D semantic segmentation.
Traditional 3D scene understanding approaches rely on labeled 3D datasets to train a model for a single task with supervision. We propose OpenScene, an alternative approach where a model predicts dense features for 3D scene points that are co-embedded with text and image pixels in CLIP feature space. This zero-shot approach enables task-agnostic training and open-vocabulary queries. For example, to perform SOTA zero-shot 3D semantic segmentation it first infers CLIP features for every 3D point and later classifies them based on similarities to embeddings of arbitrary class labels. More interestingly, it enables a suite of open-vocabulary scene understanding applications that have never been done before. For example, it allows a user to enter an arbitrary text query and then see a heat map indicating which parts of a scene match. Our approach is effective at identifying objects, materials, affordances, activities, and room types in complex 3D scenes, all using a single model trained without any labeled 3D data.