GaussianHeadTalk: Wobble-Free 3D Talking Heads with Audio Driven Gaussian Splatting
This addresses the need for stable, real-time talking head avatars for interactive applications, though it appears to be an incremental improvement combining existing techniques.
The paper tackles the problem of generating temporally unstable 3D talking heads from audio, which causes wobbling and artifacts in real-time applications, by proposing a method that maps Gaussian Splatting with 3D Morphable Models and transformer-based audio-driven parameter prediction to achieve competitive quantitative and qualitative performance.
Speech-driven talking heads have recently emerged and enable interactive avatars. However, real-world applications are limited, as current methods achieve high visual fidelity but slow or fast yet temporally unstable. Diffusion methods provide realistic image generation, yet struggle with oneshot settings. Gaussian Splatting approaches are real-time, yet inaccuracies in facial tracking, or inconsistent Gaussian mappings, lead to unstable outputs and video artifacts that are detrimental to realistic use cases. We address this problem by mapping Gaussian Splatting using 3D Morphable Models to generate person-specific avatars. We introduce transformer-based prediction of model parameters, directly from audio, to drive temporal consistency. From monocular video and independent audio speech inputs, our method enables generation of real-time talking head videos where we report competitive quantitative and qualitative performance.