ArtisanGS: Interactive Tools for Gaussian Splat Selection with AI and Human in the Loop
This work addresses the problem of controllable editing in 3DGS representations for users in graphics and animation, offering an incremental improvement over existing methods.
The paper tackles the challenge of extracting and editing objects from unstructured 3D Gaussian Splat (3DGS) scenes by introducing an interactive toolset for Gaussian Splat selection and segmentation, enabling user-guided local editing with a custom Video Diffusion Model without requiring additional optimization.
Representation in the family of 3D Gaussian Splats (3DGS) are growing into a viable alternative to traditional graphics for an expanding number of application, including recent techniques that facilitate physics simulation and animation. However, extracting usable objects from in-the-wild captures remains challenging and controllable editing techniques for this representation are limited. Unlike the bulk of emerging techniques, focused on automatic solutions or high-level editing, we introduce an interactive suite of tools centered around versatile Gaussian Splat selection and segmentation. We propose a fast AI-driven method to propagate user-guided 2D selection masks to 3DGS selections. This technique allows for user intervention in the case of errors and is further coupled with flexible manual selection and segmentation tools. These allow a user to achieve virtually any binary segmentation of an unstructured 3DGS scene. We evaluate our toolset against the state-of-the-art for Gaussian Splat selection and demonstrate their utility for downstream applications by developing a user-guided local editing approach, leveraging a custom Video Diffusion Model. With flexible selection tools, users have direct control over the areas that the AI can modify. Our selection and editing tools can be used for any in-the-wild capture without additional optimization.