Explainable Disentanglement on Discrete Speech Representations for Noise-Robust ASR
This work addresses noise-robust automatic speech recognition for real-world applications, representing an incremental improvement over existing methods.
The paper tackled the problem of discrete speech representations being suboptimal in noisy environments by proposing a method to disentangle semantic content from background noise, resulting in an 82% reduction in error rate compared to Whisper and a 35% improvement over baselines on the VBDemand test set.
Discrete audio representations are gaining traction in speech modeling due to their interpretability and compatibility with large language models, but are not always optimized for noisy or real-world environments. Building on existing works that quantize Whisper embeddings for speech-to-unit modeling, we propose disentangling semantic speech content from background noise in the latent space. Our end-to-end model separates clean speech in the form of codebook tokens, while extracting interpretable noise vectors as quantization residue which are supervised via a lightweight classifier. We show that our approach improves alignment between clean/noisy speech and text, producing speech tokens that display a high degree of noiseinvariance, and improves ASR performance. Keeping Whisper frozen, we show an 82% reduction in error rate compared to Whisper, and 35% improvement over baseline methods on the VBDemand test set. Further analyses show that the learned token space generalizes well to both seen and unseen acoustic conditions.