Progressive Disentangled Representation Learning for Fine-Grained Controllable Talking Head Synthesis
This work addresses the challenge of generating realistic and controllable talking head videos for applications like virtual avatars or video editing, representing an incremental improvement over existing methods.
The paper tackles the problem of achieving fine-grained, disentangled control over multiple facial motions in one-shot talking head synthesis, resulting in high-quality speech-lip synchronization and precise control over eye gaze, blink, head pose, and emotional expression, which previous methods struggled with.
We present a novel one-shot talking head synthesis method that achieves disentangled and fine-grained control over lip motion, eye gaze&blink, head pose, and emotional expression. We represent different motions via disentangled latent representations and leverage an image generator to synthesize talking heads from them. To effectively disentangle each motion factor, we propose a progressive disentangled representation learning strategy by separating the factors in a coarse-to-fine manner, where we first extract unified motion feature from the driving signal, and then isolate each fine-grained motion from the unified feature. We introduce motion-specific contrastive learning and regressing for non-emotional motions, and feature-level decorrelation and self-reconstruction for emotional expression, to fully utilize the inherent properties of each motion factor in unstructured video data to achieve disentanglement. Experiments show that our method provides high quality speech&lip-motion synchronization along with precise and disentangled control over multiple extra facial motions, which can hardly be achieved by previous methods.