Tackling View-Dependent Semantics in 3D Language Gaussian Splatting
This work addresses a fundamental gap in 3D scene understanding for applications like robotics and AR/VR, though it is incremental as it builds on existing 3D-GS methods.
The paper tackled the problem of view-dependent semantics in 3D language Gaussian splatting, where 3D objects can have different semantics from different viewpoints, and proposed LaGa to address this by decomposing scenes into objects and aggregating multi-view semantics, achieving a +18.7% mIoU improvement over previous SOTA on the LERF-OVS dataset.
Recent advancements in 3D Gaussian Splatting (3D-GS) enable high-quality 3D scene reconstruction from RGB images. Many studies extend this paradigm for language-driven open-vocabulary scene understanding. However, most of them simply project 2D semantic features onto 3D Gaussians and overlook a fundamental gap between 2D and 3D understanding: a 3D object may exhibit various semantics from different viewpoints--a phenomenon we term view-dependent semantics. To address this challenge, we propose LaGa (Language Gaussians), which establishes cross-view semantic connections by decomposing the 3D scene into objects. Then, it constructs view-aggregated semantic representations by clustering semantic descriptors and reweighting them based on multi-view semantics. Extensive experiments demonstrate that LaGa effectively captures key information from view-dependent semantics, enabling a more comprehensive understanding of 3D scenes. Notably, under the same settings, LaGa achieves a significant improvement of +18.7% mIoU over the previous SOTA on the LERF-OVS dataset. Our code is available at: https://github.com/SJTU-DeepVisionLab/LaGa.