GLS: Geometry-aware 3D Language Gaussian Splatting
This work addresses the problem of enhancing 3D scene understanding for applications like robotics and AR/VR by integrating reconstruction and segmentation, though it appears incremental as it builds on existing 3DGS methods.
The paper tackles the joint optimization of 3D surface reconstruction and open-vocabulary segmentation by introducing GLS, a framework based on 3D Gaussian Splatting that improves sharpness and smoothness using geometric cues and CLIP features. It achieves state-of-the-art results on MuSHRoom, ScanNet++, and LERF-OVS datasets.
Recently, 3D Gaussian Splatting (3DGS) has achieved impressive performance on indoor surface reconstruction and 3D open-vocabulary segmentation. This paper presents GLS, a unified framework of 3D surface reconstruction and open-vocabulary segmentation based on 3DGS. GLS extends two fields by improving their sharpness and smoothness. For indoor surface reconstruction, we introduce surface normal prior as a geometric cue to guide the rendered normal, and use the normal error to optimize the rendered depth. For 3D open-vocabulary segmentation, we employ 2D CLIP features to guide instance features and enhance the surface smoothness, then utilize DEVA masks to maintain their view consistency. Extensive experiments demonstrate the effectiveness of jointly optimizing surface reconstruction and 3D open-vocabulary segmentation, where GLS surpasses state-of-the-art approaches of each task on MuSHRoom, ScanNet++ and LERF-OVS datasets. Project webpage: https://jiaxiongq.github.io/GLS_ProjectPage.