LESV: Language Embedded Sparse Voxel Fusion for Open-Vocabulary 3D Scene Understanding
This work addresses limitations in 3D scene understanding for applications like robotics and AR/VR, representing an incremental improvement over existing methods.
The paper tackles the problem of spatial and semantic ambiguity in open-vocabulary 3D scene understanding by introducing a framework that uses Sparse Voxel Rasterization with regularization, achieving state-of-the-art performance on benchmarks, especially for fine-grained queries.
Recent advancements in open-vocabulary 3D scene understanding heavily rely on 3D Gaussian Splatting (3DGS) to register vision-language features into 3D space. However, we identify two critical limitations in these approaches: the spatial ambiguity arising from unstructured, overlapping Gaussians which necessitates probabilistic feature registration, and the multi-level semantic ambiguity caused by pooling features over object-level masks, which dilutes fine-grained details. To address these challenges, we present a novel framework that leverages Sparse Voxel Rasterization (SVRaster) as a structured, disjoint geometry representation. By regularizing SVRaster with monocular depth and normal priors, we establish a stable geometric foundation. This enables a deterministic, confidence-aware feature registration process and suppresses the semantic bleeding artifact common in 3DGS. Furthermore, we resolve multi-level ambiguity by exploiting the emerging dense alignment properties of foundation model AM-RADIO, avoiding the computational overhead of hierarchical training methods. Our approach achieves state-of-the-art performance on Open Vocabulary 3D Object Retrieval and Point Cloud Understanding benchmarks, particularly excelling on fine-grained queries where registration methods typically fail.