SVP: Style-Enhanced Vivid Portrait Talking Head Diffusion Model
This work addresses a limitation in talking head generation for applications like digital humans and virtual reality, but it is incremental as it builds on existing diffusion models.
The paper tackled the problem of talking head generation lacking diversity and vividness due to overlooked intrinsic styles like speaking habits and facial expressions, and proposed the SVP framework that leverages style-related information to generate diverse, vivid, and high-quality videos, outperforming state-of-the-art methods.
Talking Head Generation (THG), typically driven by audio, is an important and challenging task with broad application prospects in various fields such as digital humans, film production, and virtual reality. While diffusion model-based THG methods present high quality and stable content generation, they often overlook the intrinsic style which encompasses personalized features such as speaking habits and facial expressions of a video. As consequence, the generated video content lacks diversity and vividness, thus being limited in real life scenarios. To address these issues, we propose a novel framework named Style-Enhanced Vivid Portrait (SVP) which fully leverages style-related information in THG. Specifically, we first introduce the novel probabilistic style prior learning to model the intrinsic style as a Gaussian distribution using facial expressions and audio embedding. The distribution is learned through the 'bespoked' contrastive objective, effectively capturing the dynamic style information in each video. Then we finetune a pretrained Stable Diffusion (SD) model to inject the learned intrinsic style as a controlling signal via cross attention. Experiments show that our model generates diverse, vivid, and high-quality videos with flexible control over intrinsic styles, outperforming existing state-of-the-art methods.