IF-MDM: Implicit Face Motion Diffusion Model for High-Fidelity Realtime Talking Head Generation
This addresses the problem of slow processing in high-fidelity talking head generation for applications like virtual avatars or video conferencing, though it is incremental by building on prior diffusion and implicit motion approaches.
The paper tackles high-resolution talking head generation from a single image and audio by proposing IF-MDM, which uses implicit motion to encode faces into appearance-aware latents, achieving real-time generation at 45 fps for 512x512 videos with superior performance over existing methods.
We introduce a novel approach for high-resolution talking head generation from a single image and audio input. Prior methods using explicit face models, like 3D morphable models (3DMM) and facial landmarks, often fall short in generating high-fidelity videos due to their lack of appearance-aware motion representation. While generative approaches such as video diffusion models achieve high video quality, their slow processing speeds limit practical application. Our proposed model, Implicit Face Motion Diffusion Model (IF-MDM), employs implicit motion to encode human faces into appearance-aware compressed facial latents, enhancing video generation. Although implicit motion lacks the spatial disentanglement of explicit models, which complicates alignment with subtle lip movements, we introduce motion statistics to help capture fine-grained motion information. Additionally, our model provides motion controllability to optimize the trade-off between motion intensity and visual quality during inference. IF-MDM supports real-time generation of 512x512 resolution videos at up to 45 frames per second (fps). Extensive evaluations demonstrate its superior performance over existing diffusion and explicit face models. The code will be released publicly, available alongside supplementary materials. The video results can be found on https://bit.ly/ifmdm_supplementary.