A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose
This addresses the challenge of creating flexible, generative models for human motion capture and shape estimation from uncontrolled video data, which is incremental by building on NeRFs with articulated structures.
The paper tackles the problem of learning generative neural body models from unlabelled monocular videos by extending Neural Radiance Fields (NeRFs) with a skeleton for articulated motion, enabling recovery of volumetric body shape, appearance, and pose without ground truth labels, and it improves accuracy for novel-view synthesis and motion capture on diverse datasets.
While deep learning reshaped the classical motion capture pipeline with feed-forward networks, generative models are required to recover fine alignment via iterative refinement. Unfortunately, the existing models are usually hand-crafted or learned in controlled conditions, only applicable to limited domains. We propose a method to learn a generative neural body model from unlabelled monocular videos by extending Neural Radiance Fields (NeRFs). We equip them with a skeleton to apply to time-varying and articulated motion. A key insight is that implicit models require the inverse of the forward kinematics used in explicit surface models. Our reparameterization defines spatial latent variables relative to the pose of body parts and thereby overcomes ill-posed inverse operations with an overparameterization. This enables learning volumetric body shape and appearance from scratch while jointly refining the articulated pose; all without ground truth labels for appearance, pose, or 3D shape on the input videos. When used for novel-view-synthesis and motion capture, our neural model improves accuracy on diverse datasets. Project website: https://lemonatsu.github.io/anerf/ .