Utilizing Neural Transducers for Two-Stage Text-to-Speech via Semantic Token Prediction
This work addresses efficient and high-quality speech synthesis for TTS applications, representing an incremental improvement by integrating neural transducers into a decoupled pipeline.
The authors tackled the problem of text-to-speech synthesis by proposing a two-stage framework using a neural transducer for semantic token prediction and a non-autoregressive generator, which improved speech quality and speaker similarity in zero-shot adaptive TTS, as shown by objective and subjective metrics.
We propose a novel text-to-speech (TTS) framework centered around a neural transducer. Our approach divides the whole TTS pipeline into semantic-level sequence-to-sequence (seq2seq) modeling and fine-grained acoustic modeling stages, utilizing discrete semantic tokens obtained from wav2vec2.0 embeddings. For a robust and efficient alignment modeling, we employ a neural transducer named token transducer for the semantic token prediction, benefiting from its hard monotonic alignment constraints. Subsequently, a non-autoregressive (NAR) speech generator efficiently synthesizes waveforms from these semantic tokens. Additionally, a reference speech controls temporal dynamics and acoustic conditions at each stage. This decoupled framework reduces the training complexity of TTS while allowing each stage to focus on semantic and acoustic modeling. Our experimental results on zero-shot adaptive TTS demonstrate that our model surpasses the baseline in terms of speech quality and speaker similarity, both objectively and subjectively. We also delve into the inference speed and prosody control capabilities of our approach, highlighting the potential of neural transducers in TTS frameworks.