NeRFDeformer: NeRF Transformation from a Single View via 3D Scene Flows
This addresses the challenge of scene editing in neural radiance fields for applications like virtual reality or content creation, though it appears incremental as it builds on existing NeRF frameworks.
The paper tackles the problem of automatically modifying a NeRF representation from a single view of a non-rigidly transformed scene, achieving results that outperform existing NeRF editing and diffusion-based methods.
We present a method for automatically modifying a NeRF representation based on a single observation of a non-rigid transformed version of the original scene. Our method defines the transformation as a 3D flow, specifically as a weighted linear blending of rigid transformations of 3D anchor points that are defined on the surface of the scene. In order to identify anchor points, we introduce a novel correspondence algorithm that first matches RGB-based pairs, then leverages multi-view information and 3D reprojection to robustly filter false positives in two steps. We also introduce a new dataset for exploring the problem of modifying a NeRF scene through a single observation. Our dataset ( https://github.com/nerfdeformer/nerfdeformer ) contains 113 synthetic scenes leveraging 47 3D assets. We show that our proposed method outperforms NeRF editing methods as well as diffusion-based methods, and we also explore different methods for filtering correspondences.