ATL-Diff: Audio-Driven Talking Head Generation with Early Landmarks-Guide Noise Diffusion
This work solves the problem of generating realistic talking heads from audio for applications like virtual assistants and education, with incremental improvements in synchronization and efficiency.
The paper tackles audio-driven talking head generation by addressing synchronization limitations and reducing noise and computational costs, achieving state-of-the-art performance on MEAD and CREMA-D datasets with near real-time processing.
Audio-driven talking head generation requires precise synchronization between facial animations and audio signals. This paper introduces ATL-Diff, a novel approach addressing synchronization limitations while reducing noise and computational costs. Our framework features three key components: a Landmark Generation Module converting audio to facial landmarks, a Landmarks-Guide Noise approach that decouples audio by distributing noise according to landmarks, and a 3D Identity Diffusion network preserving identity characteristics. Experiments on MEAD and CREMA-D datasets demonstrate that ATL-Diff outperforms state-of-the-art methods across all metrics. Our approach achieves near real-time processing with high-quality animations, computational efficiency, and exceptional preservation of facial nuances. This advancement offers promising applications for virtual assistants, education, medical communication, and digital platforms. The source code is available at: \href{https://github.com/sonvth/ATL-Diff}{https://github.com/sonvth/ATL-Diff}