Video2StyleGAN: Disentangling Local and Global Variations in a Video
This work addresses video manipulation for facial editing, enabling more realistic and controlled reenactment, though it is incremental as it builds on existing StyleGAN methods.
The paper tackled the problem of applying StyleGAN-based image editing to videos by achieving fine-grained disentangled control over face location, pose, and local expressions, resulting in clear improvements over alternative approaches in challenging scenarios.
Image editing using a pretrained StyleGAN generator has emerged as a powerful paradigm for facial editing, providing disentangled controls over age, expression, illumination, etc. However, the approach cannot be directly adopted for video manipulations. We hypothesize that the main missing ingredient is the lack of fine-grained and disentangled control over face location, face pose, and local facial expressions. In this work, we demonstrate that such a fine-grained control is indeed achievable using pretrained StyleGAN by working across multiple (latent) spaces (namely, the positional space, the W+ space, and the S space) and combining the optimization results across the multiple spaces. Building on this enabling component, we introduce Video2StyleGAN that takes a target image and driving video(s) to reenact the local and global locations and expressions from the driving video in the identity of the target image. We evaluate the effectiveness of our method over multiple challenging scenarios and demonstrate clear improvements over alternative approaches.