Generative Lifting of Multiview to 3D from Unknown Pose: Wrapping NeRF inside Diffusion
This addresses the need for 3D scene modeling from unannotated 2D images, which is incremental by integrating diffusion models with NeRF for pose estimation.
The paper tackles the problem of multiview 3D reconstruction from unknown camera poses by learning a Neural Radiance Field (NeRF) and a pose prediction network simultaneously, using a denoising diffusion model to drive training, resulting in successful NeRF construction for challenging scenes where other methods fail.
We cast multiview reconstruction from unknown pose as a generative modeling problem. From a collection of unannotated 2D images of a scene, our approach simultaneously learns both a network to predict camera pose from 2D image input, as well as the parameters of a Neural Radiance Field (NeRF) for the 3D scene. To drive learning, we wrap both the pose prediction network and NeRF inside a Denoising Diffusion Probabilistic Model (DDPM) and train the system via the standard denoising objective. Our framework requires the system accomplish the task of denoising an input 2D image by predicting its pose and rendering the NeRF from that pose. Learning to denoise thus forces the system to concurrently learn the underlying 3D NeRF representation and a mapping from images to camera extrinsic parameters. To facilitate the latter, we design a custom network architecture to represent pose as a distribution, granting implicit capacity for discovering view correspondences when trained end-to-end for denoising alone. This technique allows our system to successfully build NeRFs, without pose knowledge, for challenging scenes where competing methods fail. At the conclusion of training, our learned NeRF can be extracted and used as a 3D scene model; our full system can be used to sample novel camera poses and generate novel-view images.