Faces that Speak: Jointly Synthesising Talking Face and Speech from Text
It addresses the challenge of multimodal synthesis for applications like virtual avatars or assistive technologies, though it appears incremental as it builds on existing TFG and TTS methods.
This work tackles the problem of jointly generating natural talking faces and speech from text by integrating Talking Face Generation and Text-to-Speech systems into a unified framework, achieving effective synthesis that accurately matches input text and generalizes to unseen identities.
The goal of this work is to simultaneously generate natural talking faces and speech outputs from text. We achieve this by integrating Talking Face Generation (TFG) and Text-to-Speech (TTS) systems into a unified framework. We address the main challenges of each task: (1) generating a range of head poses representative of real-world scenarios, and (2) ensuring voice consistency despite variations in facial motion for the same identity. To tackle these issues, we introduce a motion sampler based on conditional flow matching, which is capable of high-quality motion code generation in an efficient way. Moreover, we introduce a novel conditioning method for the TTS system, which utilises motion-removed features from the TFG model to yield uniform speech outputs. Our extensive experiments demonstrate that our method effectively creates natural-looking talking faces and speech that accurately match the input text. To our knowledge, this is the first effort to build a multimodal synthesis system that can generalise to unseen identities.