MPE-TTS: Customized Emotion Zero-Shot Text-To-Speech Using Multi-Modal Prompt
This work addresses the problem of inflexibility in emotion control for zero-shot TTS systems, offering a domain-specific incremental improvement.
The paper tackles the limitation of single-prompt zero-shot text-to-speech systems by proposing a multi-modal prompt approach for customized emotion generation, resulting in improved naturalness and similarity over existing systems as shown in experiments.
Most existing Zero-Shot Text-To-Speech(ZS-TTS) systems generate the unseen speech based on single prompt, such as reference speech or text descriptions, which limits their flexibility. We propose a customized emotion ZS-TTS system based on multi-modal prompt. The system disentangles speech into the content, timbre, emotion and prosody, allowing emotion prompts to be provided as text, image or speech. To extract emotion information from different prompts, we propose a multi-modal prompt emotion encoder. Additionally, we introduce an prosody predictor to fit the distribution of prosody and propose an emotion consistency loss to preserve emotion information in the predicted prosody. A diffusion-based acoustic model is employed to generate the target mel-spectrogram. Both objective and subjective experiments demonstrate that our system outperforms existing systems in terms of naturalness and similarity. The samples are available at https://mpetts-demo.github.io/mpetts_demo/.