ARTalk: Speech-Driven 3D Head Animation via Autoregressive Model
This work addresses the need for efficient and realistic 3D talking avatars for applications like virtual reality or gaming, though it is incremental as it builds on prior methods to improve speed.
The paper tackles the problem of slow generation speed in speech-driven 3D facial animation by introducing an autoregressive model that achieves real-time generation of lip movements, head poses, and eye blinks, outperforming existing methods in lip synchronization accuracy and perceived quality.
Speech-driven 3D facial animation aims to generate realistic lip movements and facial expressions for 3D head models from arbitrary audio clips. Although existing diffusion-based methods are capable of producing natural motions, their slow generation speed limits their application potential. In this paper, we introduce a novel autoregressive model that achieves real-time generation of highly synchronized lip movements and realistic head poses and eye blinks by learning a mapping from speech to a multi-scale motion codebook. Furthermore, our model can adapt to unseen speaking styles, enabling the creation of 3D talking avatars with unique personal styles beyond the identities seen during training. Extensive evaluations and user studies demonstrate that our method outperforms existing approaches in lip synchronization accuracy and perceived quality.