3DiFACE: Synthesizing and Editing Holistic 3D Facial Animation
This addresses the problem for animators and creators needing efficient, controllable 3D facial animation tools, representing a novel method for a known bottleneck.
The paper tackles the challenge of creating and editing personalized 3D facial animations from speech, which is complex and lacks diversity in existing methods, by introducing 3DiFACE, a method that generates diverse lip and head motions from audio and allows editing via keyframing, achieving control between high fidelity and diversity.
Creating personalized 3D animations with precise control and realistic head motions remains challenging for current speech-driven 3D facial animation methods. Editing these animations is especially complex and time consuming, requires precise control and typically handled by highly skilled animators. Most existing works focus on controlling style or emotion of the synthesized animation and cannot edit/regenerate parts of an input animation. They also overlook the fact that multiple plausible lip and head movements can match the same audio input. To address these challenges, we present 3DiFACE, a novel method for holistic speech-driven 3D facial animation. Our approach produces diverse plausible lip and head motions for a single audio input and allows for editing via keyframing and interpolation. Specifically, we propose a fully-convolutional diffusion model that can leverage the viseme-level diversity in our training corpus. Additionally, we employ a speaking-style personalization and a novel sparsely-guided motion diffusion to enable precise control and editing. Through quantitative and qualitative evaluations, we demonstrate that our method is capable of generating and editing diverse holistic 3D facial animations given a single audio input, with control between high fidelity and diversity. Code and models are available here: https://balamuruganthambiraja.github.io/3DiFACE