GLDiTalker: Speech-Driven 3D Facial Animation with Graph Latent Diffusion Transformer
This work improves virtual human modeling for applications like augmented reality by enhancing lip-sync accuracy and motion diversity, though it appears incremental as it builds on existing diffusion-based models.
The paper tackled the problem of speech-driven 3D facial animation by addressing modality inconsistencies that reduce motion diversity and lip-sync accuracy, proposing GLDiTalker which outperformed existing methods on standard benchmarks.
Speech-driven talking head generation is a critical yet challenging task with applications in augmented reality and virtual human modeling. While recent approaches using autoregressive and diffusion-based models have achieved notable progress, they often suffer from modality inconsistencies, particularly misalignment between audio and mesh, leading to reduced motion diversity and lip-sync accuracy. To address this, we propose GLDiTalker, a novel speech-driven 3D facial animation model based on a Graph Latent Diffusion Transformer. GLDiTalker resolves modality misalignment by diffusing signals within a quantized spatiotemporal latent space. It employs a two-stage training pipeline: the Graph-Enhanced Quantized Space Learning Stage ensures lip-sync accuracy, while the Space-Time Powered Latent Diffusion Stage enhances motion diversity. Together, these stages enable GLDiTalker to generate realistic, temporally stable 3D facial animations. Extensive evaluations on standard benchmarks demonstrate that GLDiTalker outperforms existing methods, achieving superior results in both lip-sync accuracy and motion diversity.