SEED: Speaker Embedding Enhancement Diffusion Model
This work addresses a key challenge for deploying speaker recognition in real-world applications, offering a practical solution without requiring speaker labels or changes to existing pipelines, though it is incremental as it builds on diffusion models.
The paper tackles performance degradation in speaker recognition systems due to environmental mismatch by proposing a diffusion-based method that refines speaker embeddings, improving recognition accuracy by up to 19.6% over baselines in mismatch scenarios while maintaining performance in conventional settings.
A primary challenge when deploying speaker recognition systems in real-world applications is performance degradation caused by environmental mismatch. We propose a diffusion-based method that takes speaker embeddings extracted from a pre-trained speaker recognition model and generates refined embeddings. For training, our approach progressively adds Gaussian noise to both clean and noisy speaker embeddings extracted from clean and noisy speech, respectively, via forward process of a diffusion model, and then reconstructs them to clean embeddings in the reverse process. While inferencing, all embeddings are regenerated via diffusion process. Our method needs neither speaker label nor any modification to the existing speaker recognition pipeline. Experiments on evaluation sets simulating environment mismatch scenarios show that our method can improve recognition accuracy by up to 19.6% over baseline models while retaining performance on conventional scenarios. We publish our code here https://github.com/kaistmm/seed-pytorch