Imaginary Voice: Face-styled Diffusion Model for Text-to-Speech
This addresses the problem of generating personalized speech from faces for applications like virtual assistants or accessibility, representing a novel approach by using face images as a condition for TTS for the first time.
The paper tackles zero-shot text-to-speech synthesis by generating speech with styles and voices learned from facial characteristics, achieving this through a diffusion model trained on the LRS3 dataset without extra fine-tuning for unseen speakers.
The goal of this work is zero-shot text-to-speech synthesis, with speaking styles and voices learnt from facial characteristics. Inspired by the natural fact that people can imagine the voice of someone when they look at his or her face, we introduce a face-styled diffusion text-to-speech (TTS) model within a unified framework learnt from visible attributes, called Face-TTS. This is the first time that face images are used as a condition to train a TTS model. We jointly train cross-model biometrics and TTS models to preserve speaker identity between face images and generated speech segments. We also propose a speaker feature binding loss to enforce the similarity of the generated and the ground truth speech segments in speaker embedding space. Since the biometric information is extracted directly from the face image, our method does not require extra fine-tuning steps to generate speech from unseen and unheard speakers. We train and evaluate the model on the LRS3 dataset, an in-the-wild audio-visual corpus containing background noise and diverse speaking styles. The project page is https://facetts.github.io.