TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splatting
This work addresses the challenge of synthesizing realistic 3D talking heads for applications like virtual avatars or video generation, representing an incremental improvement by simplifying the learning task through decomposition into face and mouth branches.
The paper tackled the problem of distortions in dynamic regions when synthesizing 3D talking heads by introducing TalkingGaussian, a deformation-based radiance fields framework that uses Gaussian Splatting to apply smooth deformations to persistent primitives, resulting in high-fidelity lip-synchronized videos with better facial fidelity and higher efficiency compared to previous methods.
Radiance fields have demonstrated impressive performance in synthesizing lifelike 3D talking heads. However, due to the difficulty in fitting steep appearance changes, the prevailing paradigm that presents facial motions by directly modifying point appearance may lead to distortions in dynamic regions. To tackle this challenge, we introduce TalkingGaussian, a deformation-based radiance fields framework for high-fidelity talking head synthesis. Leveraging the point-based Gaussian Splatting, facial motions can be represented in our method by applying smooth and continuous deformations to persistent Gaussian primitives, without requiring to learn the difficult appearance change like previous methods. Due to this simplification, precise facial motions can be synthesized while keeping a highly intact facial feature. Under such a deformation paradigm, we further identify a face-mouth motion inconsistency that would affect the learning of detailed speaking motions. To address this conflict, we decompose the model into two branches separately for the face and inside mouth areas, therefore simplifying the learning tasks to help reconstruct more accurate motion and structure of the mouth region. Extensive experiments demonstrate that our method renders high-quality lip-synchronized talking head videos, with better facial fidelity and higher efficiency compared with previous methods.