EmoFace: Emotion-Content Disentangled Speech-Driven 3D Talking Face Animation
This work solves the problem of generating more realistic 3D talking faces with synchronized emotions for applications in animation and virtual reality, representing an incremental improvement over prior methods.
The paper tackles the problem of speech-driven 3D talking face animation by addressing the neglect of emotion-facial motion correlations in existing methods, proposing EmoFace with a two-stream network and Mesh Attention mechanism, achieving state-of-the-art performance with metrics like 4.8863×10^-5mm for LVE on their dataset.
The creation of increasingly vivid 3D talking face has become a hot topic in recent years. Currently, most speech-driven works focus on lip synchronisation but neglect to effectively capture the correlations between emotions and facial motions. To address this problem, we propose a two-stream network called EmoFace, which consists of an emotion branch and a content branch. EmoFace employs a novel Mesh Attention mechanism to analyse and fuse the emotion features and content features. Particularly, a newly designed spatio-temporal graph-based convolution, SpiralConv3D, is used in Mesh Attention to learn potential temporal and spatial feature dependencies between mesh vertices. In addition, to the best of our knowledge, it is the first time to introduce a new self-growing training scheme with intermediate supervision to dynamically adjust the ratio of groundtruth adopted in the 3D face animation task. Comprehensive quantitative and qualitative evaluations on our high-quality 3D emotional facial animation dataset, 3D-RAVDESS ($4.8863\times 10^{-5}$mm for LVE and $0.9509\times 10^{-5}$mm for EVE), together with the public dataset VOCASET ($2.8669\times 10^{-5}$mm for LVE and $0.4664\times 10^{-5}$mm for EVE), demonstrate that our approach achieves state-of-the-art performance.