Fine-grained style control in Transformer-based Text-to-speech Synthesis
This work addresses the need for more precise style manipulation in speech synthesis, which is incremental as it builds on existing TransformerTTS methods.
The paper tackled the problem of achieving fine-grained style control in transformer-based text-to-speech synthesis by modeling speaking style with local style tokens and using cross-attention blocks for fusion, resulting in improved naturalness, intelligibility, and style transferability.
In this paper, we present a novel architecture to realize fine-grained style control on the transformer-based text-to-speech synthesis (TransformerTTS). Specifically, we model the speaking style by extracting a time sequence of local style tokens (LST) from the reference speech. The existing content encoder in TransformerTTS is then replaced by our designed cross-attention blocks for fusion and alignment between content and style. As the fusion is performed along with the skip connection, our cross-attention block provides a good inductive bias to gradually infuse the phoneme representation with a given style. Additionally, we prevent the style embedding from encoding linguistic content by randomly truncating LST during training and using wav2vec 2.0 features. Experiments show that with fine-grained style control, our system performs better in terms of naturalness, intelligibility, and style transferability. Our code and samples are publicly available.