HeadCraft: Modeling High-Detail Shape Variations for Animated 3DMMs
This addresses the challenge of detailed and animatable 3D head modeling for applications like animation and VR, representing an incremental improvement over existing neural representation methods.
The paper tackles the problem of constructing high-fidelity 3D head models with explicit animation control and detail preservation from partial observations, introducing HeadCraft, a generative model that achieves this by combining a parametric head model with learned vertex displacements, demonstrated through unconditional sampling, scan fitting, and editing.
Current advances in human head modeling allow the generation of plausible-looking 3D head models via neural representations, such as NeRFs and SDFs. Nevertheless, constructing complete high-fidelity head models with explicitly controlled animation remains an issue. Furthermore, completing the head geometry based on a partial observation, e.g., coming from a depth sensor, while preserving a high level of detail is often problematic for the existing methods. We introduce a generative model for detailed 3D head meshes on top of an articulated 3DMM, simultaneously allowing explicit animation and high-detail preservation. Our method is trained in two stages. First, we register a parametric head model with vertex displacements to each mesh of the recently introduced NPHM dataset of accurate 3D head scans. The estimated displacements are baked into a hand-crafted UV layout. Second, we train a StyleGAN model to generalize over the UV maps of displacements, which we later refer to as HeadCraft. The decomposition of the parametric model and high-quality vertex displacements allows us to animate the model and modify the regions semantically. We demonstrate the results of unconditional sampling, fitting to a scan and editing. The project page is available at https://seva100.github.io/headcraft.