StyleVideoGAN: A Temporal Generative Model using a Pretrained StyleGAN
This addresses the challenge of video synthesis for portrait generation with reduced data and computational resources, though it is incremental as it builds on existing StyleGAN capabilities.
The paper tackles the problem of generating high-quality videos with limited training data by separating spatial and temporal synthesis, using a pretrained StyleGAN for frames and training a temporal model on latent codes. The result is a model that can generate portrait videos after training on only 10 minutes of footage for 6 hours, applicable to any subject embeddable in StyleGAN.
Generative adversarial models (GANs) continue to produce advances in terms of the visual quality of still images, as well as the learning of temporal correlations. However, few works manage to combine these two interesting capabilities for the synthesis of video content: Most methods require an extensive training dataset to learn temporal correlations, while being rather limited in the resolution and visual quality of their output. We present a novel approach to the video synthesis problem that helps to greatly improve visual quality and drastically reduce the amount of training data and resources necessary for generating videos. Our formulation separates the spatial domain, in which individual frames are synthesized, from the temporal domain, in which motion is generated. For the spatial domain we use a pre-trained StyleGAN network, the latent space of which allows control over the appearance of the objects it was trained for. The expressive power of this model allows us to embed our training videos in the StyleGAN latent space. Our temporal architecture is then trained not on sequences of RGB frames, but on sequences of StyleGAN latent codes. The advantageous properties of the StyleGAN space simplify the discovery of temporal correlations. We demonstrate that it suffices to train our temporal architecture on only 10 minutes of footage of 1 subject for about 6 hours. After training, our model can not only generate new portrait videos for the training subject, but also for any random subject which can be embedded in the StyleGAN space.