Towards Lightweight and Stable Zero-shot TTS with Self-distilled Representation Disentanglement
This addresses deployment and security concerns for personalized voice cloning, but appears incremental as it builds on existing zero-shot TTS methods.
The paper tackles the problem of high deployment costs and data security in zero-shot TTS by proposing a lightweight and stable system, achieving computational efficiency with RTFs of 0.13 on CPU and 0.012 on GPU.
Zero-shot Text-To-Speech (TTS) synthesis shows great promise for personalized voice customization through voice cloning. However, current methods for achieving zero-shot TTS heavily rely on large model scales and extensive training datasets to ensure satisfactory performance and generalizability across various speakers. This raises concerns regarding both deployment costs and data security. In this paper, we present a lightweight and stable zero-shot TTS system. We introduce a novel TTS architecture designed to effectively model linguistic content and various speaker attributes from source speech and prompt speech, respectively. Furthermore, we present a two-stage self-distillation framework that constructs parallel data pairs for effectively disentangling linguistic content and speakers from the perspective of training data. Extensive experiments show that our system exhibits excellent performance and superior stability on the zero-shot TTS tasks. Moreover, it shows markedly superior computational efficiency, with RTFs of 0.13 and 0.012 on the CPU and GPU, respectively.