Two-Step Sound Source Separation: Training on Learned Latent Targets
This is an incremental improvement for audio processing researchers, offering a generalizable method to enhance neural network-based separation systems.
The paper tackles sound source separation by proposing a two-step training procedure that first learns an optimal latent space for masking-based separation and then trains a separation module in that space, achieving better performance than joint training systems.
In this paper, we propose a two-step training procedure for source separation via a deep neural network. In the first step we learn a transform (and it's inverse) to a latent space where masking-based separation performance using oracles is optimal. For the second step, we train a separation module that operates on the previously learned space. In order to do so, we also make use of a scale-invariant signal to distortion ratio (SI-SDR) loss function that works in the latent space, and we prove that it lower-bounds the SI-SDR in the time domain. We run various sound separation experiments that show how this approach can obtain better performance as compared to systems that learn the transform and the separation module jointly. The proposed methodology is general enough to be applicable to a large class of neural network end-to-end separation systems.