Online Language Splatting
This addresses the need for dynamic and interactive AI applications by overcoming the computational limitations of prior offline methods, though it is incremental as it builds on existing 3DGS-SLAM and language integration techniques.
The paper tackles the problem of enabling AI agents to interact with 3D environments and human language by developing Online Language Splatting, a framework that achieves online, near real-time, open-vocabulary language mapping within a 3DGS-SLAM system without pre-generated language features, resulting in surpassing state-of-the-art offline methods in accuracy and achieving more than 40x efficiency boost.
To enable AI agents to interact seamlessly with both humans and 3D environments, they must not only perceive the 3D world accurately but also align human language with 3D spatial representations. While prior work has made significant progress by integrating language features into geometrically detailed 3D scene representations using 3D Gaussian Splatting (GS), these approaches rely on computationally intensive offline preprocessing of language features for each input image, limiting adaptability to new environments. In this work, we introduce Online Language Splatting, the first framework to achieve online, near real-time, open-vocabulary language mapping within a 3DGS-SLAM system without requiring pre-generated language features. The key challenge lies in efficiently fusing high-dimensional language features into 3D representations while balancing the computation speed, memory usage, rendering quality and open-vocabulary capability. To this end, we innovatively design: (1) a high-resolution CLIP embedding module capable of generating detailed language feature maps in 18ms per frame, (2) a two-stage online auto-encoder that compresses 768-dimensional CLIP features to 15 dimensions while preserving open-vocabulary capabilities, and (3) a color-language disentangled optimization approach to improve rendering quality. Experimental results show that our online method not only surpasses the state-of-the-art offline methods in accuracy but also achieves more than 40x efficiency boost, demonstrating the potential for dynamic and interactive AI applications.