NeRF-In: Free-Form NeRF Inpainting with RGB-D Priors
This addresses the challenge of intuitive 3D scene editing for users working with NeRF models, representing an incremental improvement over existing methods.
The paper tackles the problem of editing pre-trained Neural Radiance Fields (NeRF) by enabling users to remove or retouch unwanted objects in 3D scenes without category-specific data or training, achieving visually plausible and structurally consistent results across multiple views with reduced time and manual effort.
Though Neural Radiance Field (NeRF) demonstrates compelling novel view synthesis results, it is still unintuitive to edit a pre-trained NeRF because the neural network's parameters and the scene geometry/appearance are often not explicitly associated. In this paper, we introduce the first framework that enables users to remove unwanted objects or retouch undesired regions in a 3D scene represented by a pre-trained NeRF without any category-specific data and training. The user first draws a free-form mask to specify a region containing unwanted objects over a rendered view from the pre-trained NeRF. Our framework first transfers the user-provided mask to other rendered views and estimates guiding color and depth images within these transferred masked regions. Next, we formulate an optimization problem that jointly inpaints the image content in all masked regions across multiple views by updating the NeRF model's parameters. We demonstrate our framework on diverse scenes and show it obtained visual plausible and structurally consistent results across multiple views using shorter time and less user manual efforts.