Emotional Talking Head Generation based on Memory-Sharing and Attention-Augmented Networks
This work addresses the limitation of existing talking head generation methods that lack emotional expression, which is important for applications in virtual avatars and human-computer interaction, though it is incremental in improving emotion modeling.
The paper tackles the problem of generating talking head videos that synchronize lip movements with audio while also reproducing the target person's facial expressions, achieving superior performance over previous methods in qualitative and quantitative experiments.
Given an audio clip and a reference face image, the goal of the talking head generation is to generate a high-fidelity talking head video. Although some audio-driven methods of generating talking head videos have made some achievements in the past, most of them only focused on lip and audio synchronization and lack the ability to reproduce the facial expressions of the target person. To this end, we propose a talking head generation model consisting of a Memory-Sharing Emotion Feature extractor (MSEF) and an Attention-Augmented Translator based on U-net (AATU). Firstly, MSEF can extract implicit emotional auxiliary features from audio to estimate more accurate emotional face landmarks.~Secondly, AATU acts as a translator between the estimated landmarks and the photo-realistic video frames. Extensive qualitative and quantitative experiments have shown the superiority of the proposed method to the previous works. Codes will be made publicly available.