End-to-End Adversarial Text-to-Speech
This work addresses the problem of simplifying text-to-speech synthesis pipelines for researchers and practitioners by eliminating multi-stage training, though it is incremental as it builds on existing adversarial and alignment techniques.
The paper tackles the challenge of synthesizing speech from text or phonemes in an end-to-end manner, achieving a mean opinion score exceeding 4 on a 5-point scale, comparable to state-of-the-art multi-stage models.
Modern text-to-speech synthesis pipelines typically involve multiple processing stages, each of which is designed or learnt independently from the rest. In this work, we take on the challenging task of learning to synthesise speech from normalised text or phonemes in an end-to-end manner, resulting in models which operate directly on character or phoneme input sequences and produce raw speech audio outputs. Our proposed generator is feed-forward and thus efficient for both training and inference, using a differentiable alignment scheme based on token length prediction. It learns to produce high fidelity audio through a combination of adversarial feedback and prediction losses constraining the generated audio to roughly match the ground truth in terms of its total duration and mel-spectrogram. To allow the model to capture temporal variation in the generated audio, we employ soft dynamic time warping in the spectrogram-based prediction loss. The resulting model achieves a mean opinion score exceeding 4 on a 5 point scale, which is comparable to the state-of-the-art models relying on multi-stage training and additional supervision.