AudioFace: Language-Assisted Speech-Driven Facial Animation with Multimodal Language Models
This work addresses the problem of inaccurate mouth articulation in speech-driven facial animation by leveraging linguistic priors, offering a novel approach for animators and virtual avatar systems.
AudioFace introduces a language-assisted framework for speech-driven facial animation that uses transcript and phoneme cues to improve mouth-related coefficient prediction, achieving superior performance across multiple metrics.
Speech-driven facial animation requires accurate correspondence between acoustic signals and facial motion, especially for articulation-related mouth movements. However, directly mapping speech audio to facial coefficients often overlooks the linguistic and phonetic structure underlying speech production. In this paper, we propose AudioFace, a language-assisted framework for speech-driven blendshape generation that treats mouth-related facial coefficient prediction as a structured generation problem guided by linguistic and articulatory information. Instead of relying solely on acoustic features, our method leverages the prior knowledge of multimodal large language models and introduces transcript- and phoneme-level cues to bridge speech signals with interpretable facial actions. Extensive experiments show that AudioFace achieves superior performance across multiple evaluation metrics, validating the effectiveness of language-assisted and multimodal-prior-guided speech-driven facial animation.