VisualSpeaker: Visually-Guided 3D Avatar Lip Synthesis
This work addresses the need for high-fidelity 3D avatar lip synthesis in human-computer interaction and accessibility, with incremental improvements over prior methods.
The paper tackled the problem of generating realistic 3D facial animations for avatars by bridging the gap between 2D visual innovations and 3D mesh methods, resulting in a 56.1% improvement in Lip Vertex Error and enhanced perceptual quality on the MEAD dataset.
Realistic, high-fidelity 3D facial animations are crucial for expressive avatar systems in human-computer interaction and accessibility. Although prior methods show promising quality, their reliance on the mesh domain limits their ability to fully leverage the rapid visual innovations seen in 2D computer vision and graphics. We propose VisualSpeaker, a novel method that bridges this gap using photorealistic differentiable rendering, supervised by visual speech recognition, for improved 3D facial animation. Our contribution is a perceptual lip-reading loss, derived by passing photorealistic 3D Gaussian Splatting avatar renders through a pre-trained Visual Automatic Speech Recognition model during training. Evaluation on the MEAD dataset demonstrates that VisualSpeaker improves both the standard Lip Vertex Error metric by 56.1% and the perceptual quality of the generated animations, while retaining the controllability of mesh-driven animation. This perceptual focus naturally supports accurate mouthings, essential cues that disambiguate similar manual signs in sign language avatars.