SegSplat: Feed-forward Gaussian Splatting and Open-Set Semantic Segmentation
This work addresses the need for practical, on-the-fly generation of semantically aware 3D environments for applications like robotic interaction and augmented reality, representing a significant step forward but not a paradigm shift.
The paper tackles the problem of combining fast 3D reconstruction with open-set semantic segmentation by introducing SegSplat, which achieves geometric fidelity comparable to state-of-the-art feed-forward 3D Gaussian Splatting methods and enables robust semantic segmentation without per-scene optimization.
We have introduced SegSplat, a novel framework designed to bridge the gap between rapid, feed-forward 3D reconstruction and rich, open-vocabulary semantic understanding. By constructing a compact semantic memory bank from multi-view 2D foundation model features and predicting discrete semantic indices alongside geometric and appearance attributes for each 3D Gaussian in a single pass, SegSplat efficiently imbues scenes with queryable semantics. Our experiments demonstrate that SegSplat achieves geometric fidelity comparable to state-of-the-art feed-forward 3D Gaussian Splatting methods while simultaneously enabling robust open-set semantic segmentation, crucially \textit{without} requiring any per-scene optimization for semantic feature integration. This work represents a significant step towards practical, on-the-fly generation of semantically aware 3D environments, vital for advancing robotic interaction, augmented reality, and other intelligent systems.