From Tokens to Faces: Investigating Discrete Speech Representations for 3D Facial Animation
For researchers in speech-driven facial animation, this work provides a systematic comparison of speech representations, but the findings are incremental as they confirm the importance of phonetic encoding already known in the field.
This paper evaluates four speech representation families for 3D facial animation, finding that phonetic-aware representations (semantic and label-based) yield comparable facial animation quality. It introduces an AVTTS pipeline using discrete representations for joint speech and facial motion decoding.
The choice of speech representation is critical in speech-driven 3D facial animation. Representations differ in what they encode: SSL features emphasize segmental and semantic cues, neural codecs yield latents optimized for acoustic reconstruction, and ASR-style objectives produce label-based spaces. We evaluate four speech representation families for 3D facial synthesis, comparing their facial reconstruction quality across two facial decoders using objective metrics and a perceptual evaluation. We additionally conduct probing analyses that relate tokenized representations to phonetic units and to articulatory deformations. We found that encoding phonetic classes is beneficial for accurate facial animation prediction on both semantic and label-based representations with comparable facial animation quality. From the latter, we introduce an Audio Visual Text-to-Speech (AVTTS) pipeline that leverages, as a shared space, discrete representations to decode speech and 3D facial motion.