High-Quality Sound Separation Across Diverse Categories via Visually-Guided Generative Modeling
This addresses the problem of high-quality sound separation across diverse categories for audio-visual applications, representing a novel methodological approach rather than an incremental improvement.
The paper tackles audio-visual sound source separation by proposing DAVIS, a generative framework that synthesizes separated sound spectrograms from noise using diffusion models and flow matching. Results show both DAVIS variants surpass existing methods in separation quality on AVE and MUSIC datasets.
We propose DAVIS, a Diffusion-based Audio-VIsual Separation framework that solves the audio-visual sound source separation task through generative learning. Existing methods typically frame sound separation as a mask-based regression problem, achieving significant progress. However, they face limitations in capturing the complex data distribution required for high-quality separation of sounds from diverse categories. In contrast, DAVIS circumvents these issues by leveraging potent generative modeling paradigms, specifically Denoising Diffusion Probabilistic Models (DDPM) and the more recent Flow Matching (FM), integrated within a specialized Separation U-Net architecture. Our framework operates by synthesizing the desired separated sound spectrograms directly from a noise distribution, conditioned concurrently on the mixed audio input and associated visual information. The inherent nature of its generative objective makes DAVIS particularly adept at producing high-quality sound separations for diverse sound categories. We present comparative evaluations of DAVIS, encompassing both its DDPM and Flow Matching variants, against leading methods on the standard AVE and MUSIC datasets. The results affirm that both variants surpass existing approaches in separation quality, highlighting the efficacy of our generative framework for tackling the audio-visual source separation task.