Talking Head Generation with Audio and Speech Related Facial Action Units
This work addresses the task of generating realistic talking head videos for applications like virtual avatars or video editing, but it is incremental as it builds on existing methods by incorporating AU information.
The paper tackled the problem of talking head generation by synthesizing lip-synchronized videos from face images and audio, using audio and speech-related facial action units (AUs) to improve mouth movement accuracy, resulting in outperforming existing methods in image quality and lip-sync accuracy.
The task of talking head generation is to synthesize a lip synchronized talking head video by inputting an arbitrary face image and audio clips. Most existing methods ignore the local driving information of the mouth muscles. In this paper, we propose a novel recurrent generative network that uses both audio and speech-related facial action units (AUs) as the driving information. AU information related to the mouth can guide the movement of the mouth more accurately. Since speech is highly correlated with speech-related AUs, we propose an Audio-to-AU module in our system to predict the speech-related AU information from speech. In addition, we use AU classifier to ensure that the generated images contain correct AU information. Frame discriminator is also constructed for adversarial training to improve the realism of the generated face. We verify the effectiveness of our model on the GRID dataset and TCD-TIMIT dataset. We also conduct an ablation study to verify the contribution of each component in our model. Quantitative and qualitative experiments demonstrate that our method outperforms existing methods in both image quality and lip-sync accuracy.