Towards Robust FastSpeech 2 by Modelling Residual Multimodality
This addresses audio quality issues in text-to-speech for expressive datasets, representing an incremental improvement.
The paper tackled the problem of audio distortions in FastSpeech 2 for expressive speech synthesis, caused by over-smooth mel-spectrogram predictions from MSE loss, and introduced TVC-GMM to model residual multimodality, which reduced spectrogram smoothness and improved perceptual audio quality as shown by objective and subjective evaluations.
State-of-the-art non-autoregressive text-to-speech (TTS) models based on FastSpeech 2 can efficiently synthesise high-fidelity and natural speech. For expressive speech datasets however, we observe characteristic audio distortions. We demonstrate that such artefacts are introduced to the vocoder reconstruction by over-smooth mel-spectrogram predictions, which are induced by the choice of mean-squared-error (MSE) loss for training the mel-spectrogram decoder. With MSE loss FastSpeech 2 is limited to learn conditional averages of the training distribution, which might not lie close to a natural sample if the distribution still appears multimodal after all conditioning signals. To alleviate this problem, we introduce TVC-GMM, a mixture model of Trivariate-Chain Gaussian distributions, to model the residual multimodality. TVC-GMM reduces spectrogram smoothness and improves perceptual audio quality in particular for expressive datasets as shown by both objective and subjective evaluation.