Instant Colorization of Gaussian Splats
This work provides a fast, closed-form solution for back-projecting 2D data onto 3D Gaussian splats, benefiting applications in scene editing and segmentation for the 3D Gaussian Splatting community.
The paper introduces a method for efficiently mapping 2D image information (e.g., color, features) back onto 3D Gaussian splats, enabling tasks like scene relighting and semantic segmentation. It achieves up to 10x speedup over gradient descent baselines.
Gaussian Splatting has recently become one of the most popular frameworks for photorealistic 3D scene reconstruction and rendering. While current rasterizers allow for efficient mappings of 3D Gaussian splats onto 2D camera views, this work focuses on mapping 2D image information (e.g. color, neural features or segmentation masks) efficiently back onto an existing scene of Gaussian splats. This 'opposite' direction enables applications ranging from scene relighting and stylization to 3D semantic segmentation, but also introduces challenges, such as view-dependent colorization and occlusion handling. Our approach tackles these challenges using the normal equation to solve a visibility-weighted least squares problem for every Gaussian and can be implemented efficiently with existing differentiable rasterizers. We demonstrate the effectiveness of our approach on scene relighting, feature enrichment and 3D semantic segmentation tasks, achieving up to an order of magnitude speedup compared to gradient descent-based baselines.