Depth-supervised NeRF: Fewer Views and Faster Training for Free
This addresses a common failure mode in NeRF for 3D scene reconstruction, offering a cheap and effective solution to improve performance with limited data, though it is incremental as it builds on existing NeRF methods.
The paper tackles the problem of Neural Radiance Fields (NeRF) failing to fit correct geometries with insufficient input views by introducing DS-NeRF, a depth-supervised loss that uses sparse 3D points from structure-from-motion as free depth supervision, resulting in better image rendering with fewer views and 2-3x faster training.
A commonly observed failure mode of Neural Radiance Field (NeRF) is fitting incorrect geometries when given an insufficient number of input views. One potential reason is that standard volumetric rendering does not enforce the constraint that most of a scene's geometry consist of empty space and opaque surfaces. We formalize the above assumption through DS-NeRF (Depth-supervised Neural Radiance Fields), a loss for learning radiance fields that takes advantage of readily-available depth supervision. We leverage the fact that current NeRF pipelines require images with known camera poses that are typically estimated by running structure-from-motion (SFM). Crucially, SFM also produces sparse 3D points that can be used as "free" depth supervision during training: we add a loss to encourage the distribution of a ray's terminating depth matches a given 3D keypoint, incorporating depth uncertainty. DS-NeRF can render better images given fewer training views while training 2-3x faster. Further, we show that our loss is compatible with other recently proposed NeRF methods, demonstrating that depth is a cheap and easily digestible supervisory signal. And finally, we find that DS-NeRF can support other types of depth supervision such as scanned depth sensors and RGB-D reconstruction outputs.