EditEmoTalk: Controllable Speech-Driven 3D Facial Animation with Continuous Expression Editing
This work addresses the need for continuous and fine-grained emotional control in speech-driven facial animation, which is incremental by improving upon existing methods that rely on discrete emotion categories.
The paper tackles the problem of generating realistic 3D facial animations from speech with limited emotional control, presenting EditEmoTalk, which achieves superior controllability, expressiveness, and generalization while maintaining accurate lip synchronization.
Speech-driven 3D facial animation aims to generate realistic and expressive facial motions directly from audio. While recent methods achieve high-quality lip synchronization, they often rely on discrete emotion categories, limiting continuous and fine-grained emotional control. We present EditEmoTalk, a controllable speech-driven 3D facial animation framework with continuous emotion editing. The key idea is a boundary-aware semantic embedding that learns the normal directions of inter-emotion decision boundaries, enabling a continuous expression manifold for smooth emotion manipulation. Moreover, we introduce an emotional consistency loss that enforces semantic alignment between the generated motion dynamics and the target emotion embedding through a mapping network, ensuring faithful emotional expression. Extensive experiments demonstrate that EditEmoTalk achieves superior controllability, expressiveness, and generalization while maintaining accurate lip synchronization. Code and pretrained models will be released.