Contrastive Training with LLM-generated Near-Misses for Robust Code-Switching Speech Recognition
For ASR systems handling code-switching, this method offers a robust improvement over standard fine-tuning, though the gains are incremental.
The paper tackles code-switching speech recognition by proposing a contrastive training framework that uses LLM-generated near-miss hypotheses at code-switching regions, achieving over 2% reduction in error rates on two datasets.
Code-switching (CS), the alternation between multiple languages within a single utterance, remains challenging for Automatic Speech Recognition (ASR). To address this issue, we propose a Point-of-Interest (POI)-aware contrastive training framework that improves recognition at CS-critical regions. We first identify CS spans by adopting POI detection method from literature, then construct acoustically plausible near-miss hypotheses by perturbing POIs in ASR N-best outputs and expanding candidates with a large language model. Hard but plausible negatives are retained through filtering with acoustic, phonemic, and textual constraints. Finally, we fine-tune Whisper-small with LoRA using a POI-weighted cross-entropy anchor objective together with a multi-negative contrastive ranking loss. Experiments on CS-FLEURS (cmn-eng) and ViMedCSS (vie-eng) show consistent reductions of over 2% in both general and CS-aware error rates compared to standard LoRA fine-tuning.