Reference-guided Controllable Inpainting of Neural Radiance Fields
This addresses the need for editing tools in NeRF-based view synthesis, offering a practical solution for users to modify scenes, though it is incremental as it builds on existing NeRF and inpainting methods.
The paper tackles the problem of editing Neural Radiance Fields (NeRFs) by inpainting regions in a view-consistent and controllable way, achieving superior performance to baselines with the advantage of user control via a single inpainted image.
The popularity of Neural Radiance Fields (NeRFs) for view synthesis has led to a desire for NeRF editing tools. Here, we focus on inpainting regions in a view-consistent and controllable manner. In addition to the typical NeRF inputs and masks delineating the unwanted region in each view, we require only a single inpainted view of the scene, i.e., a reference view. We use monocular depth estimators to back-project the inpainted view to the correct 3D positions. Then, via a novel rendering technique, a bilateral solver can construct view-dependent effects in non-reference views, making the inpainted region appear consistent from any view. For non-reference disoccluded regions, which cannot be supervised by the single reference view, we devise a method based on image inpainters to guide both the geometry and appearance. Our approach shows superior performance to NeRF inpainting baselines, with the additional advantage that a user can control the generated scene via a single inpainted image. Project page: https://ashmrz.github.io/reference-guided-3d