Semantic Consistent Language Gaussian Splatting for Point-Level Open-vocabulary Querying
This addresses open-vocabulary 3D scene understanding for robotics applications like natural language-driven manipulation, representing an incremental improvement over existing methods.
The paper tackles the problem of inconsistent 2D mask supervision and lack of robust 3D point-level retrieval in open-vocabulary 3D scene understanding for robotics, achieving mIoU improvements of +4.14, +20.42, and +1.7 on three benchmark datasets.
Open-vocabulary 3D scene understanding is crucial for robotics applications, such as natural language-driven manipulation, human-robot interaction, and autonomous navigation. Existing methods for querying 3D Gaussian Splatting often struggle with inconsistent 2D mask supervision and lack a robust 3D point-level retrieval mechanism. In this work, (i) we present a novel point-level querying framework that performs tracking on segmentation masks to establish a semantically consistent ground-truth for distilling the language Gaussians; (ii) we introduce a GT-anchored querying approach that first retrieves the distilled ground-truth and subsequently uses the ground-truth to query the individual Gaussians. Extensive experiments on three benchmark datasets demonstrate that the proposed method outperforms state-of-the-art performance. Our method achieves an mIoU improvement of +4.14, +20.42, and +1.7 on the LERF, 3D-OVS, and Replica datasets. These results validate our framework as a promising step toward open-vocabulary understanding in real-world robotic systems.