StyleFaceV: Face Video Generation via Decomposing and Recomposing Pretrained StyleGAN3
This addresses the challenge of realistic face video synthesis for computer vision and graphics applications, representing an incremental improvement over existing methods.
The paper tackles the problem of generating realistic face videos with stable identity and natural movements by proposing StyleFaceV, a framework that decomposes and recomposes appearance and pose in StyleGAN3's latent space, achieving state-of-the-art results in high-fidelity 1024x1024 video generation without requiring high-resolution training videos.
Realistic generative face video synthesis has long been a pursuit in both computer vision and graphics community. However, existing face video generation methods tend to produce low-quality frames with drifted facial identities and unnatural movements. To tackle these challenges, we propose a principled framework named StyleFaceV, which produces high-fidelity identity-preserving face videos with vivid movements. Our core insight is to decompose appearance and pose information and recompose them in the latent space of StyleGAN3 to produce stable and dynamic results. Specifically, StyleGAN3 provides strong priors for high-fidelity facial image generation, but the latent space is intrinsically entangled. By carefully examining its latent properties, we propose our decomposition and recomposition designs which allow for the disentangled combination of facial appearance and movements. Moreover, a temporal-dependent model is built upon the decomposed latent features, and samples reasonable sequences of motions that are capable of generating realistic and temporally coherent face videos. Particularly, our pipeline is trained with a joint training strategy on both static images and high-quality video data, which is of higher data efficiency. Extensive experiments demonstrate that our framework achieves state-of-the-art face video generation results both qualitatively and quantitatively. Notably, StyleFaceV is capable of generating realistic $1024\times1024$ face videos even without high-resolution training videos.