Multi-Speaker End-to-End Speech Synthesis
This work addresses multi-speaker speech synthesis for applications like virtual assistants, but it is incremental as it builds on an existing model.
The paper tackles the problem of generating high-fidelity speech from multiple speakers by extending ClariNet, a fully end-to-end text-to-wave model, to incorporate low-dimensional trainable speaker embeddings shared across components, resulting in outperforming state-of-the-art systems in naturalness.
In this work, we extend ClariNet (Ping et al., 2019), a fully end-to-end speech synthesis model (i.e., text-to-wave), to generate high-fidelity speech from multiple speakers. To model the unique characteristic of different voices, low dimensional trainable speaker embeddings are shared across each component of ClariNet and trained together with the rest of the model. We demonstrate that the multi-speaker ClariNet outperforms state-of-the-art systems in terms of naturalness, because the whole model is jointly optimized in an end-to-end manner.