Lifting by Gaussians: A Simple, Fast and Flexible Method for 3D Instance Segmentation
This method addresses the problem of efficient and flexible 3D instance segmentation for 3DGS reconstructions, enabling applications like novel view synthesis and asset extraction without retraining, though it is incremental as it builds on existing 2D segmentation and 3DGS techniques.
The paper tackles 3D instance segmentation in open-world scenes by introducing Lifting By Gaussians (LBG), which lifts 2D segmentation masks and features onto 3D Gaussian Splatted Radiance Fields without per-scene training, achieving an order of magnitude faster performance and superior results in semantic segmentation and asset extraction.
We introduce Lifting By Gaussians (LBG), a novel approach for open-world instance segmentation of 3D Gaussian Splatted Radiance Fields (3DGS). Recently, 3DGS Fields have emerged as a highly efficient and explicit alternative to Neural Field-based methods for high-quality Novel View Synthesis. Our 3D instance segmentation method directly lifts 2D segmentation masks from SAM (alternately FastSAM, etc.), together with features from CLIP and DINOv2, directly fusing them onto 3DGS (or similar Gaussian radiance fields such as 2DGS). Unlike previous approaches, LBG requires no per-scene training, allowing it to operate seamlessly on any existing 3DGS reconstruction. Our approach is not only an order of magnitude faster and simpler than existing approaches; it is also highly modular, enabling 3D semantic segmentation of existing 3DGS fields without requiring a specific parametrization of the 3D Gaussians. Furthermore, our technique achieves superior semantic segmentation for 2D semantic novel view synthesis and 3D asset extraction results while maintaining flexibility and efficiency. We further introduce a novel approach to evaluate individually segmented 3D assets from 3D radiance field segmentation methods.