kNN Retrieval for Simple and Effective Zero-Shot Multi-speaker Text-to-Speech
This enables the development of multi-speaker TTS systems for low-resource domains and languages, though it is incremental as it builds on existing SSL feature insights.
The paper tackles the problem of zero-shot multi-speaker text-to-speech (TTS) by proposing kNN-TTS, a framework that uses retrieval methods based on self-supervised learning (SSL) features, achieving performance comparable to state-of-the-art models while requiring training data from only a single speaker.
While recent zero-shot multi-speaker text-to-speech (TTS) models achieve impressive results, they typically rely on extensive transcribed speech datasets from numerous speakers and intricate training pipelines. Meanwhile, self-supervised learning (SSL) speech features have emerged as effective intermediate representations for TTS. Further, SSL features from different speakers that are linearly close share phonetic information while maintaining individual speaker identity. In this study, we introduce kNN-TTS, a simple and effective framework for zero-shot multi-speaker TTS using retrieval methods which leverage the linear relationships between SSL features. Objective and subjective evaluations show that our models, trained on transcribed speech from a single speaker only, achieve performance comparable to state-of-the-art models that are trained on significantly larger training datasets. The low training data requirements mean that kNN-TTS is well suited for the development of multi-speaker TTS systems for low-resource domains and languages. We also introduce an interpolation parameter which enables fine-grained voice morphing. Demo samples are available at https://idiap.github.io/knn-tts