Instant3dit: Multiview Inpainting for Fast Editing of 3D Objects
This addresses the need for fast, high-quality 3D editing for users in graphics and AI, representing a significant speed improvement from hours to seconds.
The paper tackles the problem of slow 3D object editing by proposing a generative technique that edits 3D shapes in approximately 3 seconds, achieving higher-quality results compared to previous methods.
We propose a generative technique to edit 3D shapes, represented as meshes, NeRFs, or Gaussian Splats, in approximately 3 seconds, without the need for running an SDS type of optimization. Our key insight is to cast 3D editing as a multiview image inpainting problem, as this representation is generic and can be mapped back to any 3D representation using the bank of available Large Reconstruction Models. We explore different fine-tuning strategies to obtain both multiview generation and inpainting capabilities within the same diffusion model. In particular, the design of the inpainting mask is an important factor of training an inpainting model, and we propose several masking strategies to mimic the types of edits a user would perform on a 3D shape. Our approach takes 3D generative editing from hours to seconds and produces higher-quality results compared to previous works.