UniTTS: Residual Learning of Unified Embedding Space for Speech Style Control
This addresses the challenge of cleanly separating multiple style attributes in speech synthesis for applications like voice assistants or audiobooks, representing an incremental improvement over existing methods.
The paper tackles the problem of controlling overlapping style attributes like speaker ID and emotion in expressive speech synthesis by proposing UniTTS, which learns a unified embedding space using residuals to avoid interference, resulting in high-fidelity synthesized speech with harmonious attribute separation.
We propose a novel high-fidelity expressive speech synthesis model, UniTTS, that learns and controls overlapping style attributes avoiding interference. UniTTS represents multiple style attributes in a single unified embedding space by the residuals between the phoneme embeddings before and after applying the attributes. The proposed method is especially effective in controlling multiple attributes that are difficult to separate cleanly, such as speaker ID and emotion, because it minimizes redundancy when adding variance in speaker ID and emotion, and additionally, predicts duration, pitch, and energy based on the speaker ID and emotion. In experiments, the visualization results exhibit that the proposed methods learned multiple attributes harmoniously in a manner that can be easily separated again. As well, UniTTS synthesized high-fidelity speech signals controlling multiple style attributes. The synthesized speech samples are presented at https://anonymous-authors2022.github.io/paper_works/UniTTS/demos/.