GOV-NeSF: Generalizable Open-Vocabulary Neural Semantic Fields
This work addresses the problem of generalizable 3D scene understanding for applications like robotics and AR/VR, representing a novel method for a known bottleneck.
The paper tackles the limited generalizability of open-vocabulary 3D scene understanding methods by introducing GOV-NeSF, which achieves state-of-the-art performance in 2D and 3D semantic segmentation without requiring ground truth labels or depth priors, and generalizes across scenes and datasets without fine-tuning.
Recent advancements in vision-language foundation models have significantly enhanced open-vocabulary 3D scene understanding. However, the generalizability of existing methods is constrained due to their framework designs and their reliance on 3D data. We address this limitation by introducing Generalizable Open-Vocabulary Neural Semantic Fields (GOV-NeSF), a novel approach offering a generalizable implicit representation of 3D scenes with open-vocabulary semantics. We aggregate the geometry-aware features using a cost volume, and propose a Multi-view Joint Fusion module to aggregate multi-view features through a cross-view attention mechanism, which effectively predicts view-specific blending weights for both colors and open-vocabulary features. Remarkably, our GOV-NeSF exhibits state-of-the-art performance in both 2D and 3D open-vocabulary semantic segmentation, eliminating the need for ground truth semantic labels or depth priors, and effectively generalize across scenes and datasets without fine-tuning.