Trace3D: Consistent Segmentation Lifting via Gaussian Instance Tracing
This addresses the challenge of noisy 3D segmentation for applications like scene editing, though it appears incremental as it builds on existing Gaussian Splatting methods.
The paper tackles the problem of inconsistent 2D segmentation masks when lifting to 3D in Gaussian Splatting, introducing Gaussian Instance Tracing (GIT) to correct inconsistencies and an adaptive density control mechanism, resulting in sharper 2D and 3D segmentation boundaries and clean 3D asset extraction.
We address the challenge of lifting 2D visual segmentation to 3D in Gaussian Splatting. Existing methods often suffer from inconsistent 2D masks across viewpoints and produce noisy segmentation boundaries as they neglect these semantic cues to refine the learned Gaussians. To overcome this, we introduce Gaussian Instance Tracing (GIT), which augments the standard Gaussian representation with an instance weight matrix across input views. Leveraging the inherent consistency of Gaussians in 3D, we use this matrix to identify and correct 2D segmentation inconsistencies. Furthermore, since each Gaussian ideally corresponds to a single object, we propose a GIT-guided adaptive density control mechanism to split and prune ambiguous Gaussians during training, resulting in sharper and more coherent 2D and 3D segmentation boundaries. Experimental results show that our method extracts clean 3D assets and consistently improves 3D segmentation in both online (e.g., self-prompting) and offline (e.g., contrastive lifting) settings, enabling applications such as hierarchical segmentation, object extraction, and scene editing.