Generative Speech Enhancement Based on Cloned Networks
This addresses speech enhancement for noisy audio signals, offering a generative approach that reduces artifacts, but it is incremental as it builds on existing generative methods like WaveNet.
The paper tackles speech enhancement by regenerating clean speech from noise-robust salient features extracted via weight-sharing clones trained on noisy versions of the same speech, achieving state-of-the-art performance in generative enhancers with statistically significant improvements across SNR ranges.
We propose to implement speech enhancement by the regeneration of clean speech from a salient representation extracted from the noisy signal. The network that extracts salient features is trained using a set of weight-sharing clones of the extractor network. The clones receive mel-frequency spectra of different noisy versions of the same speech signal as input. By encouraging the outputs of the clones to be similar for these different input signals, we train a feature extractor network that is robust to noise. At inference, the salient features form the input to a WaveNet network that generates a natural and clean speech signal with the same attributes as the ground-truth clean signal. As the signal becomes noisier, our system produces natural sounding errors that stay on the speech manifold, in place of traditional artifacts found in other systems. Our experiments confirm that our generative enhancement system provides state-of-the-art enhancement performance within the generative class of enhancers according to a MUSHRA-like test. The clones based system matches or outperforms the other systems at each input signal-to-noise (SNR) range with statistical significance.