NeRFuser: Large-Scale Scene Representation by NeRF Fusion
This addresses the need for practical tools to operate on implicit visual data structures like NeRFs, enabling more efficient scene manipulation without access to original image sets, though it is incremental as it builds on existing NeRF methods.
The paper tackled the problem of extending image-based vision techniques like registration and blending to Neural Radiance Fields (NeRFs) for large-scale scene representation, proposing NeRFuser, which achieved robustness on public benchmarks and a self-collected dataset, including for challenging views.
A practical benefit of implicit visual representations like Neural Radiance Fields (NeRFs) is their memory efficiency: large scenes can be efficiently stored and shared as small neural nets instead of collections of images. However, operating on these implicit visual data structures requires extending classical image-based vision techniques (e.g., registration, blending) from image sets to neural fields. Towards this goal, we propose NeRFuser, a novel architecture for NeRF registration and blending that assumes only access to pre-generated NeRFs, and not the potentially large sets of images used to generate them. We propose registration from re-rendering, a technique to infer the transformation between NeRFs based on images synthesized from individual NeRFs. For blending, we propose sample-based inverse distance weighting to blend visual information at the ray-sample level. We evaluate NeRFuser on public benchmarks and a self-collected object-centric indoor dataset, showing the robustness of our method, including to views that are challenging to render from the individual source NeRFs.