Learning to regulate 3D head shape by removing occluding hair from in-the-wild images
This addresses a specific challenge in 3D head reconstruction for animation or avatar creation, offering an incremental improvement over existing methods.
The paper tackles the problem of reconstructing the upper head shape in 3D face models, which is often occluded by hair in in-the-wild images, by introducing a method to remove occluding hair and reconstruct skin, achieving state-of-the-art results on popular benchmarks.
Recent 3D face reconstruction methods reconstruct the entire head compared to earlier approaches which only model the face. Although these methods accurately reconstruct facial features, they do not explicitly regulate the upper part of the head. Extracting information about this part of the head is challenging due to varying degrees of occlusion by hair. We present a novel approach for modeling the upper head by removing occluding hair and reconstructing the skin, revealing information about the head shape. We introduce three objectives: 1) a dice consistency loss that enforces similarity between the overall head shape of the source and rendered image, 2) a scale consistency loss to ensure that head shape is accurately reproduced even if the upper part of the head is not visible, and 3) a 71 landmark detector trained using a moving average loss function to detect additional landmarks on the head. These objectives are used to train an encoder in an unsupervised manner to regress FLAME parameters from in-the-wild input images. Our unsupervised 3DMM model achieves state-of-the-art results on popular benchmarks and can be used to infer the head shape, facial features, and textures for direct use in animation or avatar creation.