Emotional Conversation: Empowering Talking Faces with Cohesive Expression, Gaze and Pose Generation
This work solves the problem of synthesizing emotionally coherent talking faces for applications in film and game production, representing an incremental advancement over prior methods.
The paper tackles the problem of generating realistic talking faces by addressing the alignment of emotion with facial cues like expression, gaze, and pose, which existing methods often ignore, and demonstrates significant improvements in visual quality and emotional alignment on the MEAD dataset.
Vivid talking face generation holds immense potential applications across diverse multimedia domains, such as film and game production. While existing methods accurately synchronize lip movements with input audio, they typically ignore crucial alignments between emotion and facial cues, which include expression, gaze, and head pose. These alignments are indispensable for synthesizing realistic videos. To address these issues, we propose a two-stage audio-driven talking face generation framework that employs 3D facial landmarks as intermediate variables. This framework achieves collaborative alignment of expression, gaze, and pose with emotions through self-supervised learning. Specifically, we decompose this task into two key steps, namely speech-to-landmarks synthesis and landmarks-to-face generation. The first step focuses on simultaneously synthesizing emotionally aligned facial cues, including normalized landmarks that represent expressions, gaze, and head pose. These cues are subsequently reassembled into relocated facial landmarks. In the second step, these relocated landmarks are mapped to latent key points using self-supervised learning and then input into a pretrained model to create high-quality face images. Extensive experiments on the MEAD dataset demonstrate that our model significantly advances the state-of-the-art performance in both visual quality and emotional alignment.