ReferSplat: Referring Segmentation in 3D Gaussian Splatting
This work addresses a crucial challenge for advancing embodied AI by enabling 3D multi-modal understanding with natural language, though it is incremental as it builds on existing 3D Gaussian splatting and segmentation methods.
The paper tackles the problem of segmenting target objects in 3D Gaussian scenes based on natural language descriptions, which often include spatial relationships or attributes, and proposes ReferSplat, a framework that achieves state-of-the-art performance on this new task and 3D open-vocabulary segmentation benchmarks.
We introduce Referring 3D Gaussian Splatting Segmentation (R3DGS), a new task that aims to segment target objects in a 3D Gaussian scene based on natural language descriptions, which often contain spatial relationships or object attributes. This task requires the model to identify newly described objects that may be occluded or not directly visible in a novel view, posing a significant challenge for 3D multi-modal understanding. Developing this capability is crucial for advancing embodied AI. To support research in this area, we construct the first R3DGS dataset, Ref-LERF. Our analysis reveals that 3D multi-modal understanding and spatial relationship modeling are key challenges for R3DGS. To address these challenges, we propose ReferSplat, a framework that explicitly models 3D Gaussian points with natural language expressions in a spatially aware paradigm. ReferSplat achieves state-of-the-art performance on both the newly proposed R3DGS task and 3D open-vocabulary segmentation benchmarks. Dataset and code are available at https://github.com/heshuting555/ReferSplat.