Unleashing the Power of Natural Audio Featuring Multiple Sound Sources
This work addresses the generalization challenge for sound separation in natural audio scenarios, which is critical for artificial auditory perception, though it appears incremental by building on existing methods with new training strategies.
The paper tackles the problem of universal sound separation in real-world audio by proposing ClearSep, a framework that uses a data engine to decompose naturally mixed audio into independent tracks, achieving state-of-the-art performance across multiple tasks.
Universal sound separation aims to extract clean audio tracks corresponding to distinct events from mixed audio, which is critical for artificial auditory perception. However, current methods heavily rely on artificially mixed audio for training, which limits their ability to generalize to naturally mixed audio collected in real-world environments. To overcome this limitation, we propose ClearSep, an innovative framework that employs a data engine to decompose complex naturally mixed audio into multiple independent tracks, thereby allowing effective sound separation in real-world scenarios. We introduce two remix-based evaluation metrics to quantitatively assess separation quality and use these metrics as thresholds to iteratively apply the data engine alongside model training, progressively optimizing separation performance. In addition, we propose a series of training strategies tailored to these separated independent tracks to make the best use of them. Extensive experiments demonstrate that ClearSep achieves state-of-the-art performance across multiple sound separation tasks, highlighting its potential for advancing sound separation in natural audio scenarios. For more examples and detailed results, please visit our demo page at https://clearsep.github.io.