From Oracle to Noisy Context: Mitigating Contextual Exposure Bias in Speech-LLMs
This addresses robustness issues in contextual ASR for speech processing applications, but is incremental as it builds on existing methods like DPO and Whisper.
The paper tackles contextual exposure bias in Speech-LLMs for ASR, where training with oracle history leads to performance drops under realistic error-prone contexts, and proposes a training framework that reduces WER from 5.59% to 5.17% on TED-LIUM 3 with two-utterance history.
Contextual automatic speech recognition (ASR) with Speech-LLMs is typically trained with oracle conversation history, but relies on error-prone history at inference, causing a train-test mismatch in the context channel that we term contextual exposure bias. We propose a unified training framework to improve robustness under realistic histories: (i) Teacher Error Knowledge by using Whisper large-v3 hypotheses as training-time history, (ii) Context Dropout to regularize over-reliance on history, and (iii) Direct Preference Optimization (DPO) on curated failure cases. Experiments on TED-LIUM 3 (in-domain) and zero-shot LibriSpeech (out-of-domain) show consistent gains under predicted-history decoding. With a two-utterance history as context, SFT with Whisper hypotheses reduce WER from 5.59% (oracle-history training) to 5.47%, and DPO further improves to 5.17%. Under irrelevant-context attacks, DPO yields the smallest degradation (5.17% -> 5.63%), indicating improved robustness to misleading context. Our code and models are published on https://github.com/XYGuo1996/Contextual_Speech_LLMs.