Stitch it in Time: GAN-Based Facial Editing of Real Videos
This work addresses the problem of temporally coherent facial video editing for applications in video production and manipulation, representing an incremental advancement over existing methods.
The paper tackles the challenge of maintaining temporal consistency when applying GAN-based facial editing to videos, proposing a framework that leverages StyleGAN's natural alignment and neural networks' tendency to learn low-frequency functions to provide a consistent prior. The method demonstrates significant improvements over state-of-the-art approaches, producing meaningful face manipulations with higher temporal consistency on challenging talking head videos.
The ability of Generative Adversarial Networks to encode rich semantics within their latent space has been widely adopted for facial image editing. However, replicating their success with videos has proven challenging. Sets of high-quality facial videos are lacking, and working with videos introduces a fundamental barrier to overcome - temporal coherency. We propose that this barrier is largely artificial. The source video is already temporally coherent, and deviations from this state arise in part due to careless treatment of individual components in the editing pipeline. We leverage the natural alignment of StyleGAN and the tendency of neural networks to learn low frequency functions, and demonstrate that they provide a strongly consistent prior. We draw on these insights and propose a framework for semantic editing of faces in videos, demonstrating significant improvements over the current state-of-the-art. Our method produces meaningful face manipulations, maintains a higher degree of temporal consistency, and can be applied to challenging, high quality, talking head videos which current methods struggle with.