Rethinking Speaker Embeddings for Speech Generation: Sub-Center Modeling for Capturing Intra-Speaker Diversity
This work addresses a specific bottleneck in personalized speech generation for applications like voice conversion, offering an incremental improvement over existing methods.
The paper tackled the problem of speaker embeddings losing intra-speaker variation when optimized for speaker recognition, making them suboptimal for speech generation. They proposed a sub-center modeling approach that improved naturalness and prosodic expressiveness in synthesized speech for voice conversion tasks.
Modeling the rich prosodic variations inherent in human speech is essential for generating natural-sounding speech. While speaker embeddings are commonly used as conditioning inputs in personalized speech generation, they are typically optimized for speaker recognition, which encourages the loss of intra-speaker variation. This strategy makes them suboptimal for speech generation in terms of modeling the rich variations at the output speech distribution. In this work, we propose a novel speaker embedding network that employs multiple sub-centers per speaker class during training, instead of a single center as in conventional approaches. This sub-center modeling allows the embedding to capture a broader range of speaker-specific variations while maintaining speaker classification performance. We demonstrate the effectiveness of the proposed embeddings on a voice conversion task, showing improved naturalness and prosodic expressiveness in the synthesized speech.