Video2StyleGAN: Encoding Video in Latent Space for Manipulation
This work addresses the challenge of consistent and efficient face video manipulation for applications in video editing and generation, representing a domain-specific advancement.
The paper tackles the problem of applying GAN-based face editing to videos by ensuring temporal consistency, real-time speed, and accurate encoding of facial motions and expressions, achieving real-time performance at 66 fps and outperforming existing single-image methods.
Many recent works have been proposed for face image editing by leveraging the latent space of pretrained GANs. However, few attempts have been made to directly apply them to videos, because 1) they do not guarantee temporal consistency, 2) their application is limited by their processing speed on videos, and 3) they cannot accurately encode details of face motion and expression. To this end, we propose a novel network to encode face videos into the latent space of StyleGAN for semantic face video manipulation. Based on the vision transformer, our network reuses the high-resolution portion of the latent vector to enforce temporal consistency. To capture subtle face motions and expressions, we design novel losses that involve sparse facial landmarks and dense 3D face mesh. We have thoroughly evaluated our approach and successfully demonstrated its application to various face video manipulations. Particularly, we propose a novel network for pose/expression control in a 3D coordinate system. Both qualitative and quantitative results have shown that our approach can significantly outperform existing single image methods, while achieving real-time (66 fps) speed.