Better Pseudo-labeling with Multi-ASR Fusion and Error Correction by SpeechLLM
This addresses the challenge of error propagation and disjoint optimization in pseudo-labeling pipelines for ASR researchers and practitioners, though it appears incremental as it builds on existing multi-ASR and LLM approaches.
The paper tackles the problem of generating high-quality pseudo-labels for training ASR models by proposing a unified multi-ASR prompt-driven framework that uses LLMs for postprocessing instead of traditional voting methods, showing significant improvements in transcription accuracy and better performance in semi-supervised ASR training across datasets.
Automatic speech recognition (ASR) models rely on high-quality transcribed data for effective training. Generating pseudo-labels for large unlabeled audio datasets often relies on complex pipelines that combine multiple ASR outputs through multi-stage processing, leading to error propagation, information loss and disjoint optimization. We propose a unified multi-ASR prompt-driven framework using postprocessing by either textual or speech-based large language models (LLMs), replacing voting or other arbitration logic for reconciling the ensemble outputs. We perform a comparative study of multiple architectures with and without LLMs, showing significant improvements in transcription accuracy compared to traditional methods. Furthermore, we use the pseudo-labels generated by the various approaches to train semi-supervised ASR models for different datasets, again showing improved performance with textual and speechLLM transcriptions compared to baselines.