EmbedTalk: Triplane-Free Talking Head Synthesis using Embedding-Driven Gaussian Deformation
This work provides a more efficient and higher-quality method for real-time talking head synthesis, benefiting applications requiring low-latency and high-fidelity virtual avatars.
This paper tackles the problem of real-time talking head synthesis using deformable 3D Gaussian Splatting, which typically relies on tri-plane encoding. They introduce EmbedTalk, a method that replaces tri-plane encoding with learnt embeddings to drive speech deformations, resulting in superior rendering quality, lip synchronization, and motion consistency, while achieving over 60 FPS on a mobile GPU.
Real-time talking head synthesis increasingly relies on deformable 3D Gaussian Splatting (3DGS) due to its low latency. Tri-planes are the standard choice for encoding Gaussians prior to deformation, since they provide a continuous domain with explicit spatial relationships. However, tri-plane representations are limited by grid resolution and approximation errors introduced by projecting 3D volumetric fields onto 2D subspaces. Recent work has shown the superiority of learnt embeddings for driving temporal deformations in 4D scene reconstruction. We introduce $\textbf{EmbedTalk}$, which shows how such embeddings can be leveraged for modelling speech deformations in talking head synthesis. Through comprehensive experiments, we show that EmbedTalk outperforms existing 3DGS-based methods in rendering quality, lip synchronisation, and motion consistency, while remaining competitive with state-of-the-art generative models. Moreover, replacing the tri-plane encoding with learnt embeddings enables significantly more compact models that achieve over 60 FPS on a mobile GPU (RTX 2060 6 GB). Our code will be placed in the public domain on acceptance.