RepeaTTS: Towards Feature Discovery through Repeated Fine-Tuning
This work addresses controllability issues in text-to-speech systems for users needing precise speech synthesis, but it is incremental as it builds on existing fine-tuning methods.
The paper tackled the problem of limited and uncontrollable variation in prompt-based text-to-speech models by proposing a novel fine-tuning regime that exploits model variance to discover latent features, resulting in improved controllability for models without emotional disclosure.
A Prompt-based Text-To-Speech model allows a user to control different aspects of speech, such as speaking rate and perceived gender, through natural language instruction. Although user-friendly, such approaches are on one hand constrained: control is limited to acoustic features exposed to the model during training, and too flexible on the other: the same inputs yields uncontrollable variation that are reflected in the corpus statistics. We investigate a novel fine-tuning regime to address both of these issues at the same time by exploiting the uncontrollable variance of the model. Through principal component analysis of thousands of synthesised samples, we determine latent features that account for the highest proportion of the output variance and incorporate them as new labels for secondary fine-tuning. We evaluate the proposed methods on two models trained on an expressive Icelandic speech corpus, one with emotional disclosure and one without. In the case of the model without emotional disclosure, the method yields both continuous and discrete features that improve overall controllability of the model.