SUTA-LM: Bridging Test-Time Adaptation and Language Model Rescoring for Robust ASR
This work addresses performance drops in ASR systems due to real-world domain mismatches, offering a solution for more reliable speech recognition in varied environments, though it is incremental as it builds on existing TTA and rescoring techniques.
The paper tackled the problem of domain mismatch in automatic speech recognition (ASR) by proposing SUTA-LM, which combines test-time adaptation with language model rescoring to mitigate interference between the two methods, achieving robust results across 18 diverse ASR datasets.
Despite progress in end-to-end ASR, real-world domain mismatches still cause performance drops, which Test-Time Adaptation (TTA) aims to mitigate by adjusting models during inference. Recent work explores combining TTA with external language models, using techniques like beam search rescoring or generative error correction. In this work, we identify a previously overlooked challenge: TTA can interfere with language model rescoring, revealing the nontrivial nature of effectively combining the two methods. Based on this insight, we propose SUTA-LM, a simple yet effective extension of SUTA, an entropy-minimization-based TTA approach, with language model rescoring. SUTA-LM first applies a controlled adaptation process guided by an auto-step selection mechanism leveraging both acoustic and linguistic information, followed by language model rescoring to refine the outputs. Experiments on 18 diverse ASR datasets show that SUTA-LM achieves robust results across a wide range of domains.