TDMM-LM: Bridging Facial Understanding and Animation via Language Models
This work addresses the problem of text-guided facial animation for researchers and developers, offering a novel approach by framing it as a language modeling task, though it is incremental in applying existing methods to a new domain.
The paper tackles the lack of well-annotated text-paired facial corpora for facial animation by synthesizing a large dataset of about 80 hours of facial videos and using it to train language models for bidirectional tasks: interpreting facial motion into language and generating facial parameters from text, achieving strong generalization.
Text-guided human body animation has advanced rapidly, yet facial animation lags due to the scarcity of well-annotated, text-paired facial corpora. To close this gap, we leverage foundation generative models to synthesize a large, balanced corpus of facial behavior. We design prompts suite covering emotions and head motions, generate about 80 hours of facial videos with multiple generators, and fit per-frame 3D facial parameters, yielding large-scale (prompt and parameter) pairs for training. Building on this dataset, we probe language models for bidirectional competence over facial motion via two complementary tasks: (1) Motion2Language: given a sequence of 3D facial parameters, the model produces natural-language descriptions capturing content, style, and dynamics; and (2) Language2Motion: given a prompt, the model synthesizes the corresponding sequence of 3D facial parameters via quantized motion tokens for downstream animation. Extensive experiments show that in this setting language models can both interpret and synthesize facial motion with strong generalization. To best of our knowledge, this is the first work to cast facial-parameter modeling as a language problem, establishing a unified path for text-conditioned facial animation and motion understanding.