Multi-speaker Multi-style Text-to-speech Synthesis With Single-speaker Single-style Training Data Scenarios
This addresses a limitation in cross-speaker style transfer for text-to-speech synthesis, enabling more flexible and diverse speech generation without requiring multi-style data from individual speakers.
The paper tackles the problem of generating expressive speech by combining any styles and timbres from a multi-speaker corpus where each speaker has only one style, bypassing the need for a single speaker with multi-style recordings. The proposed method, based on Tacotron2 with a fine-grained text-based prosody module and speaker controller, successfully transfers styles between speakers, with experiments showing increased diversity by adjusting prosody features.
In the existing cross-speaker style transfer task, a source speaker with multi-style recordings is necessary to provide the style for a target speaker. However, it is hard for one speaker to express all expected styles. In this paper, a more general task, which is to produce expressive speech by combining any styles and timbres from a multi-speaker corpus in which each speaker has a unique style, is proposed. To realize this task, a novel method is proposed. This method is a Tacotron2-based framework but with a fine-grained text-based prosody predicting module and a speaker identity controller. Experiments demonstrate that the proposed method can successfully express a style of one speaker with the timber of another speaker bypassing the dependency on a single speaker's multi-style corpus. Moreover, the explicit prosody features used in the prosody predicting module can increase the diversity of synthetic speech by adjusting the value of prosody features.