Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions
This enables more intuitive 3D scene editing for applications like virtual reality and content creation, though it builds incrementally on existing diffusion models.
The paper tackles the problem of editing 3D NeRF scenes using text instructions, achieving realistic and targeted edits on large-scale, real-world scenes compared to prior work.
We propose a method for editing NeRF scenes with text-instructions. Given a NeRF of a scene and the collection of images used to reconstruct it, our method uses an image-conditioned diffusion model (InstructPix2Pix) to iteratively edit the input images while optimizing the underlying scene, resulting in an optimized 3D scene that respects the edit instruction. We demonstrate that our proposed method is able to edit large-scale, real-world scenes, and is able to accomplish more realistic, targeted edits than prior work.