Nerfbusters: Removing Ghostly Artifacts from Casually Captured NeRFs
This addresses artifacts in novel-view synthesis for casual NeRF captures, but is incremental as it builds on existing NeRF and diffusion methods.
The paper tackles the problem of ghostly artifacts like floaters and flawed geometry in Neural Radiance Fields (NeRFs) captured casually, by proposing a new dataset and evaluation procedure, and a 3D diffusion-based method that removes floaters and improves geometry.
Casually captured Neural Radiance Fields (NeRFs) suffer from artifacts such as floaters or flawed geometry when rendered outside the camera trajectory. Existing evaluation protocols often do not capture these effects, since they usually only assess image quality at every 8th frame of the training capture. To push forward progress in novel-view synthesis, we propose a new dataset and evaluation procedure, where two camera trajectories are recorded of the scene: one used for training, and the other for evaluation. In this more challenging in-the-wild setting, we find that existing hand-crafted regularizers do not remove floaters nor improve scene geometry. Thus, we propose a 3D diffusion-based method that leverages local 3D priors and a novel density-based score distillation sampling loss to discourage artifacts during NeRF optimization. We show that this data-driven prior removes floaters and improves scene geometry for casual captures.