Zero-shot text-to-speech synthesis conditioned using self-supervised speech representation model
This work addresses a gap in speaker reproduction for zero-shot TTS, offering incremental improvements for speech synthesis applications.
The paper tackles the problem of reproducing speaker characteristics for unseen speakers in zero-shot text-to-speech synthesis by proposing a method conditioned on a self-supervised speech representation model, achieving improved similarity and enabling speech-rhythm transfer as shown in evaluations.
This paper proposes a zero-shot text-to-speech (TTS) conditioned by a self-supervised speech-representation model acquired through self-supervised learning (SSL). Conventional methods with embedding vectors from x-vector or global style tokens still have a gap in reproducing the speaker characteristics of unseen speakers. A novel point of the proposed method is the direct use of the SSL model to obtain embedding vectors from speech representations trained with a large amount of data. We also introduce the separate conditioning of acoustic features and a phoneme duration predictor to obtain the disentangled embeddings between rhythm-based speaker characteristics and acoustic-feature-based ones. The disentangled embeddings will enable us to achieve better reproduction performance for unseen speakers and rhythm transfer conditioned by different speeches. Objective and subjective evaluations showed that the proposed method can synthesize speech with improved similarity and achieve speech-rhythm transfer.