SSNAPS: Audio-Visual Separation of Speech and Background Noise with Diffusion Inverse Sampling
It addresses speech separation and enhancement in noisy environments for applications like communication and scene analysis, showing strong performance gains over supervised methods.
This paper tackles audio-visual single-microphone speech separation and enhancement in real-world noise by modeling clean speech and ambient noise with diffusion priors and using generative inverse sampling. The method, which is entirely unsupervised, consistently outperforms leading supervised baselines in word error rate across mixtures of 1, 2, and 3 speakers with noise, and extends to off-screen speaker separation while producing high-fidelity noise for acoustic scene detection.
This paper addresses the challenge of audio-visual single-microphone speech separation and enhancement in the presence of real-world environmental noise. Our approach is based on generative inverse sampling, where we model clean speech and ambient noise with dedicated diffusion priors and jointly leverage them to recover all underlying sources. To achieve this, we reformulate a recent inverse sampler to match our setting. We evaluate on mixtures of 1, 2, and 3 speakers with noise and show that, despite being entirely unsupervised, our method consistently outperforms leading supervised baselines in \ac{WER} across all conditions. We further extend our framework to handle off-screen speaker separation. Moreover, the high fidelity of the separated noise component makes it suitable for downstream acoustic scene detection. Demo page: https://ssnapsicml.github.io/ssnapsicml2026/