Neural Implicit Vision-Language Feature Fields
This addresses the problem of dynamic, text-prompt-based scene segmentation for robotics or AR/VR applications, representing an incremental advance over existing 2D methods.
The paper tackles zero-shot volumetric open-vocabulary semantic scene segmentation by fusing vision-language model features into a neural implicit representation, enabling segmentation in 3D and 2D with real-time performance on noisy real-world data, as demonstrated quantitatively on the ScanNet dataset.
Recently, groundbreaking results have been presented on open-vocabulary semantic image segmentation. Such methods segment each pixel in an image into arbitrary categories provided at run-time in the form of text prompts, as opposed to a fixed set of classes defined at training time. In this work, we present a zero-shot volumetric open-vocabulary semantic scene segmentation method. Our method builds on the insight that we can fuse image features from a vision-language model into a neural implicit representation. We show that the resulting feature field can be segmented into different classes by assigning points to natural language text prompts. The implicit volumetric representation enables us to segment the scene both in 3D and 2D by rendering feature maps from any given viewpoint of the scene. We show that our method works on noisy real-world data and can run in real-time on live sensor data dynamically adjusting to text prompts. We also present quantitative comparisons on the ScanNet dataset.