Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation
This addresses the challenge of producing natural head movements and facial expressions in talking-face generation without needing additional reference videos.
The authors tackled the problem of generating realistic talking-face videos from just one identity image and audio sequence, achieving state-of-the-art results on two datasets.
Talking face generation has historically struggled to produce head movements and natural facial expressions without guidance from additional reference videos. Recent developments in diffusion-based generative models allow for more realistic and stable data synthesis and their performance on image and video generation has surpassed that of other generative models. In this work, we present an autoregressive diffusion model that requires only one identity image and audio sequence to generate a video of a realistic talking human head. Our solution is capable of hallucinating head movements, facial expressions, such as blinks, and preserving a given background. We evaluate our model on two different datasets, achieving state-of-the-art results on both of them.