Talking Head from Speech Audio using a Pre-trained Image Generator
This addresses the problem of creating realistic talking-head videos for applications like virtual avatars or video editing, though it is incremental as it builds on existing StyleGAN and encoder methods.
The paper tackles generating high-resolution talking-head videos from speech audio and a single identity image by modeling frames as points in StyleGAN's latent space and training a network to map speech to latent displacements. It significantly outperforms recent state-of-the-art methods on one dataset and achieves comparable performance on another, as measured by PSNR, SSIM, FID, and LMD metrics.
We propose a novel method for generating high-resolution videos of talking-heads from speech audio and a single 'identity' image. Our method is based on a convolutional neural network model that incorporates a pre-trained StyleGAN generator. We model each frame as a point in the latent space of StyleGAN so that a video corresponds to a trajectory through the latent space. Training the network is in two stages. The first stage is to model trajectories in the latent space conditioned on speech utterances. To do this, we use an existing encoder to invert the generator, mapping from each video frame into the latent space. We train a recurrent neural network to map from speech utterances to displacements in the latent space of the image generator. These displacements are relative to the back-projection into the latent space of an identity image chosen from the individuals depicted in the training dataset. In the second stage, we improve the visual quality of the generated videos by tuning the image generator on a single image or a short video of any chosen identity. We evaluate our model on standard measures (PSNR, SSIM, FID and LMD) and show that it significantly outperforms recent state-of-the-art methods on one of two commonly used datasets and gives comparable performance on the other. Finally, we report on ablation experiments that validate the components of the model. The code and videos from experiments can be found at https://mohammedalghamdi.github.io/talking-heads-acm-mm