Interactive Speech and Noise Modeling for Speech Enhancement
This work provides a strong specific gain in speech enhancement performance for users in noisy environments, offering improved clarity and separation.
This paper addresses the challenge of diverse background noise in speech enhancement by proposing SN-Net, a two-branch convolutional neural network that simultaneously models speech and noise. The SN-Net, which includes interaction modules between its branches, significantly outperforms state-of-the-art methods on public datasets for speech enhancement and also shows superior performance for speaker separation.
Speech enhancement is challenging because of the diversity of background noise types. Most of the existing methods are focused on modelling the speech rather than the noise. In this paper, we propose a novel idea to model speech and noise simultaneously in a two-branch convolutional neural network, namely SN-Net. In SN-Net, the two branches predict speech and noise, respectively. Instead of information fusion only at the final output layer, interaction modules are introduced at several intermediate feature domains between the two branches to benefit each other. Such an interaction can leverage features learned from one branch to counteract the undesired part and restore the missing component of the other and thus enhance their discrimination capabilities. We also design a feature extraction module, namely residual-convolution-and-attention (RA), to capture the correlations along temporal and frequency dimensions for both the speech and the noises. Evaluations on public datasets show that the interaction module plays a key role in simultaneous modeling and the SN-Net outperforms the state-of-the-art by a large margin on various evaluation metrics. The proposed SN-Net also shows superior performance for speaker separation.