Dimitra: Audio-driven Diffusion model for Expressive Talking Head Generation
This work addresses the problem of generating realistic talking heads with synchronized lip motion, facial expression, and head pose for applications in video synthesis and human-computer interaction, representing a strong specific gain in this domain.
The authors tackled audio-driven talking head generation by proposing Dimitra, a framework that uses a conditional Motion Diffusion Transformer to model facial motion sequences with 3D representation, conditioned on audio and a reference image, resulting in outperforming existing approaches on datasets like VoxCeleb2 and HDTF.
We propose Dimitra, a novel framework for audio-driven talking head generation, streamlined to learn lip motion, facial expression, as well as head pose motion. Specifically, we train a conditional Motion Diffusion Transformer (cMDT) by modeling facial motion sequences with 3D representation. We condition the cMDT with only two input signals, an audio-sequence, as well as a reference facial image. By extracting additional features directly from audio, Dimitra is able to increase quality and realism of generated videos. In particular, phoneme sequences contribute to the realism of lip motion, whereas text transcript to facial expression and head pose realism. Quantitative and qualitative experiments on two widely employed datasets, VoxCeleb2 and HDTF, showcase that Dimitra is able to outperform existing approaches for generating realistic talking heads imparting lip motion, facial expression, and head pose.