Improve Bilingual TTS Using Dynamic Language and Phonology Embedding
This work addresses the problem of improving bilingual TTS quality for monolingual speakers, though it appears incremental as it builds on existing TTS methods with specific enhancements.
The paper tackled the challenge of modeling accurate pronunciation and intonation for a second language in bilingual text-to-speech (TTS) without interference from the first language, specifically in a Mandarin-English system, and resulted in producing significantly more natural and standard spoken English speech from a monolingual Chinese speaker.
In most cases, bilingual TTS needs to handle three types of input scripts: first language only, second language only, and second language embedded in the first language. In the latter two situations, the pronunciation and intonation of the second language are usually quite different due to the influence of the first language. Therefore, it is a big challenge to accurately model the pronunciation and intonation of the second language in different contexts without mutual interference. This paper builds a Mandarin-English TTS system to acquire more standard spoken English speech from a monolingual Chinese speaker. We introduce phonology embedding to capture the English differences between different phonology. Embedding mask is applied to language embedding for distinguishing information between different languages and to phonology embedding for focusing on English expression. We specially design an embedding strength modulator to capture the dynamic strength of language and phonology. Experiments show that our approach can produce significantly more natural and standard spoken English speech of the monolingual Chinese speaker. From analysis, we find that suitable phonology control contributes to better performance in different scenarios.