Distilling LLM Semantic Priors into Encoder-Only Multi-Talker ASR with Talker-Count Routing
This work addresses the challenge of efficient and robust multi-talker speech recognition for applications like transcription and communication, representing an incremental improvement over existing methods.
The paper tackled the problem of computationally expensive and fragile LLM-based multi-talker ASR by proposing an encoder-only framework that distills LLM semantic priors into the encoder, achieving comparable performance to LLM-based systems for two-talker conditions and significant improvements for three-talker conditions with a small real-time factor.
Large language models (LLMs) provide strong semantic priors that can improve multi-talker automatic speech recognition (MT-ASR), but using an LLM as an autoregressive decoder is computationally expensive and remains fragile under heavy overlap. In this paper, we propose an encoder-only MT-ASR framework that adapts an LLM to multi-talker conditioning and distills its semantic guidance into the encoder during training, while retaining fast CTC-style decoding at inference. Our model employs a post-encoder separator with serialized CTC to produce talker-ordered transcripts, and leverages an adapted LLM-based SOT objective as a multi-talker-aware teacher signal to explicitly regularize mixed-speech representations. To further support variable numbers of talkers, we introduce a Talker-Count Head that predicts the talker count and dynamically selects the appropriate decoding branch. Experiments on LibriMix show that the proposed encoder-only model achieves comparable performance to LLM-based systems in the two-talker condition, while delivering significant improvements in the three-talker condition with significant small RTF.