Conditional Diffusion Model for Target Speaker Extraction
This addresses the problem of isolating a specific speaker's voice from background noise or other speakers for applications like hearing aids or voice assistants, representing an incremental advance in generative audio processing.
The paper tackles target speaker extraction from mixed audio by proposing DiffSpEx, a generative method based on conditional diffusion models, achieving an SI-SDR of 12.9 dB and a NISQA score of 3.56 on the WSJ0-2mix dataset, with further improvements through fine-tuning for personalization.
We propose DiffSpEx, a generative target speaker extraction method based on score-based generative modelling through stochastic differential equations. DiffSpEx deploys a continuous-time stochastic diffusion process in the complex short-time Fourier transform domain, starting from the target speaker source and converging to a Gaussian distribution centred on the mixture of sources. For the reverse-time process, a parametrised score function is conditioned on a target speaker embedding to extract the target speaker from the mixture of sources. We utilise ECAPA-TDNN target speaker embeddings and condition the score function alternately on the SDE time embedding and the target speaker embedding. The potential of DiffSpEx is demonstrated with the WSJ0-2mix dataset, achieving an SI-SDR of 12.9 dB and a NISQA score of 3.56. Moreover, we show that fine-tuning a pre-trained DiffSpEx model to a specific speaker further improves performance, enabling personalisation in target speaker extraction.