Automatic Evaluation of Speaker Similarity
This addresses the costly need for perceptual evaluations in speaker similarity assessment for speech synthesis, though it is incremental as it builds on existing speaker verification systems.
The paper tackles the problem of speaker leakage in multi-speaker text-to-speech models by proposing an automatic evaluation method for speaker similarity, achieving 0.96 accuracy and up to 0.78 Pearson correlation with human perceptual scores.
We introduce a new automatic evaluation method for speaker similarity assessment, that is consistent with human perceptual scores. Modern neural text-to-speech models require a vast amount of clean training data, which is why many solutions switch from single speaker models to solutions trained on examples from many different speakers. Multi-speaker models bring new possibilities, such as a faster creation of new voices, but also a new problem - speaker leakage, where the speaker identity of a synthesized example might not match those of the target speaker. Currently, the only way to discover this issue is through costly perceptual evaluations. In this work, we propose an automatic method for assessment of speaker similarity. For that purpose, we extend the recent work on speaker verification systems and evaluate how different metrics and speaker embeddings models reflect Multiple Stimuli with Hidden Reference and Anchor (MUSHRA) scores. Our experiments show that we can train a model to predict speaker similarity MUSHRA scores from speaker embeddings with 0.96 accuracy and significant correlation up to 0.78 Pearson score at the utterance level.