Frontend Token Enhancement for Token-Based Speech Recognition
This work addresses noise robustness in token-based ASR, which is an incremental improvement for speech processing applications.
The paper tackled the problem of environmental noise degrading token-based speech recognition by introducing a frontend system to estimate clean speech tokens from noisy speech, achieving the best performance with wave-to-token enhancement on the CHiME-4 dataset and mostly outperforming systems based on continuous SSL features.
Discretized representations of speech signals are efficient alternatives to continuous features for various speech applications, including automatic speech recognition (ASR) and speech language models. However, these representations, such as semantic or phonetic tokens derived from clustering outputs of self-supervised learning (SSL) speech models, are susceptible to environmental noise, which can degrade backend task performance. In this work, we introduce a frontend system that estimates clean speech tokens from noisy speech and evaluate it on an ASR backend using semantic tokens. We consider four types of enhancement models based on their input/output domains: wave-to-wave, token-to-token, continuous SSL features-to-token, and wave-to-token. These models are trained independently of ASR backends. Experiments on the CHiME-4 dataset demonstrate that wave-to-token enhancement achieves the best performance among the frontends. Moreover, it mostly outperforms the ASR system based on continuous SSL features.