High-fidelity Facial Avatar Reconstruction from Monocular Video with Generative Priors
This addresses the problem of realistic facial modeling for computer graphics and vision applications, but it is incremental as it builds on existing NeRF and 3D-GAN techniques.
The paper tackles high-fidelity facial avatar reconstruction from monocular video by proposing a NeRF-based method that uses a personalized generative prior from 3D-GAN, achieving superior novel view synthesis and face reenactment compared to existing works.
High-fidelity facial avatar reconstruction from a monocular video is a significant research problem in computer graphics and computer vision. Recently, Neural Radiance Field (NeRF) has shown impressive novel view rendering results and has been considered for facial avatar reconstruction. However, the complex facial dynamics and missing 3D information in monocular videos raise significant challenges for faithful facial reconstruction. In this work, we propose a new method for NeRF-based facial avatar reconstruction that utilizes 3D-aware generative prior. Different from existing works that depend on a conditional deformation field for dynamic modeling, we propose to learn a personalized generative prior, which is formulated as a local and low dimensional subspace in the latent space of 3D-GAN. We propose an efficient method to construct the personalized generative prior based on a small set of facial images of a given individual. After learning, it allows for photo-realistic rendering with novel views and the face reenactment can be realized by performing navigation in the latent space. Our proposed method is applicable for different driven signals, including RGB images, 3DMM coefficients, and audios. Compared with existing works, we obtain superior novel view synthesis results and faithfully face reenactment performance.