NeRF-Insert: 3D Local Editing with Multimodal Control Signals
This enables more flexible and precise 3D scene editing for applications in computer graphics and vision, though it builds incrementally on existing NeRF and in-painting techniques.
The paper tackles the problem of making high-quality local edits to 3D NeRF scenes by proposing NeRF-Insert, which frames editing as an in-painting problem to preserve global structure and accepts multimodal inputs like text, images, CAD models, and masks. The results show better visual quality and stronger consistency with the original NeRF compared to previous methods.
We propose NeRF-Insert, a NeRF editing framework that allows users to make high-quality local edits with a flexible level of control. Unlike previous work that relied on image-to-image models, we cast scene editing as an in-painting problem, which encourages the global structure of the scene to be preserved. Moreover, while most existing methods use only textual prompts to condition edits, our framework accepts a combination of inputs of different modalities as reference. More precisely, a user may provide a combination of textual and visual inputs including images, CAD models, and binary image masks for specifying a 3D region. We use generic image generation models to in-paint the scene from multiple viewpoints, and lift the local edits to a 3D-consistent NeRF edit. Compared to previous methods, our results show better visual quality and also maintain stronger consistency with the original NeRF.