Semantic Grouping Network for Audio Source Separation
This work addresses audio source separation for applications in audio processing and multimedia, offering a novel approach that eliminates the need for visual inputs, though it builds on existing audio-visual separation concepts.
The paper tackles the problem of audio source separation by learning to disentangle sound representations and extract high-level semantics directly from audio mixtures, without relying on visual cues. The proposed Semantic Grouping Network (SGN) significantly outperforms previous audio-only and audio-visual methods on benchmarks like MUSIC, FUSS, MUSDB18, and VGG-Sound.
Recently, audio-visual separation approaches have taken advantage of the natural synchronization between the two modalities to boost audio source separation performance. They extracted high-level semantics from visual inputs as the guidance to help disentangle sound representation for individual sources. Can we directly learn to disentangle the individual semantics from the sound itself? The dilemma is that multiple sound sources are mixed together in the original space. To tackle the difficulty, in this paper, we present a novel Semantic Grouping Network, termed as SGN, that can directly disentangle sound representations and extract high-level semantic information for each source from input audio mixture. Specifically, SGN aggregates category-wise source features through learnable class tokens of sounds. Then, the aggregated semantic features can be used as the guidance to separate the corresponding audio sources from the mixture. We conducted extensive experiments on music-only and universal sound separation benchmarks: MUSIC, FUSS, MUSDB18, and VGG-Sound. The results demonstrate that our SGN significantly outperforms previous audio-only methods and audio-visual models without utilizing additional visual cues.