SHeaP: Self-Supervised Head Geometry Predictor Learned via 2D Gaussians
This addresses the challenge of scalable 3D head modeling for visual applications, offering an incremental improvement over prior self-supervised methods.
The paper tackles the problem of 3D head reconstruction from monocular images without 3D ground truth by proposing SHeaP, a self-supervised method using 2D Gaussians, which surpasses existing approaches in geometric accuracy on benchmarks and outperforms state-of-the-art in emotion classification.
Accurate, real-time 3D reconstruction of human heads from monocular images and videos underlies numerous visual applications. As 3D ground truth data is hard to come by at scale, previous methods have sought to learn from abundant 2D videos in a self-supervised manner. Typically, this involves the use of differentiable mesh rendering, which is effective but faces limitations. To improve on this, we propose SHeaP (Self-supervised Head Geometry Predictor Learned via 2D Gaussians). Given a source image, we predict a 3DMM mesh and a set of Gaussians that are rigged to this mesh. We then reanimate this rigged head avatar to match a target frame, and backpropagate photometric losses to both the 3DMM and Gaussian prediction networks. We find that using Gaussians for rendering substantially improves the effectiveness of this self-supervised approach. Training solely on 2D data, our method surpasses existing self-supervised approaches in geometric evaluations on the NoW benchmark for neutral faces and a new benchmark for non-neutral expressions. Our method also produces highly expressive meshes, outperforming state-of-the-art in emotion classification.