H-NeRF: Neural Radiance Fields for Rendering and Temporal Reconstruction of Humans in Motion
This enables high-quality human reconstruction from limited input for applications in VR/AR and animation, representing an incremental advance combining existing techniques.
The paper tackles the problem of rendering and 4D reconstruction of humans in motion from sparse camera views or monocular video by constraining neural radiance fields with an implicit human body model, achieving robust generalization beyond training poses and views.
We present neural radiance fields for rendering and temporal (4D) reconstruction of humans in motion (H-NeRF), as captured by a sparse set of cameras or even from a monocular video. Our approach combines ideas from neural scene representation, novel-view synthesis, and implicit statistical geometric human representations, coupled using novel loss functions. Instead of learning a radiance field with a uniform occupancy prior, we constrain it by a structured implicit human body model, represented using signed distance functions. This allows us to robustly fuse information from sparse views and generalize well beyond the poses or views observed in training. Moreover, we apply geometric constraints to co-learn the structure of the observed subject -- including both body and clothing -- and to regularize the radiance field to geometrically plausible solutions. Extensive experiments on multiple datasets demonstrate the robustness and the accuracy of our approach, its generalization capabilities significantly outside a small training set of poses and views, and statistical extrapolation beyond the observed shape.