Style Description based Text-to-Speech with Conditional Prosodic Layer Normalization based Diffusion GAN
This addresses text-to-speech synthesis with style control, offering a faster method for applications like voice assistants, though it appears incremental by combining existing techniques.
The paper tackles generating high-fidelity speech from style descriptions and text, achieving results in only 4 denoising steps with strong quantitative metrics on LibriTTS and PromptSpeech datasets.
In this paper, we present a Diffusion GAN based approach (Prosodic Diff-TTS) to generate the corresponding high-fidelity speech based on the style description and content text as an input to generate speech samples within only 4 denoising steps. It leverages the novel conditional prosodic layer normalization to incorporate the style embeddings into the multi head attention based phoneme encoder and mel spectrogram decoder based generator architecture to generate the speech. The style embedding is generated by fine tuning the pretrained BERT model on auxiliary tasks such as pitch, speaking speed, emotion,gender classifications. We demonstrate the efficacy of our proposed architecture on multi-speaker LibriTTS and PromptSpeech datasets, using multiple quantitative metrics that measure generated accuracy and MOS.