Time-Frequency Weighted Losses for Phoneme Reconstruction in DNN-Based Speech Enhancement
For speech enhancement systems, this work provides a method to improve phoneme-level intelligibility, particularly for consonants, which are often poorly reconstructed.
The paper proposes a time-frequency weighting framework for DNN-based speech enhancement that modulates the SDR loss based on local speech presence, SIR, and spectral flux, improving phoneme recognition accuracy, especially for consonants, and enhancing objective frequency-weighted metrics.
Conventional training losses for speech enhancement based on the signal-to-distortion ratio (SDR) treat all time-frequency (TF) regions uniformly, overlooking the fine-grained spectral cues that are relevant to specific phoneme intelligibility. We propose a TF weighting framework that modulates the SDR objective based on local speech presence, speech-to-interference ratio (SIR), and spectral flux. By integrating these factors into a differentiable objective, the framework emphasizes TF bins with high speech-noise competition while also accounting for transient cues such as consonant bursts. Experimental results show that our approach improves objective frequency-weighted enhancement metrics, as well as phoneme recognition accuracy, particularly for consonants. Spectral analysis shows better reconstruction of mid-frequency structures at less adverse SIR.