Adapting Multi-Lingual ASR Models for Handling Multiple Talkers
This addresses the problem of handling multiple talkers in speech recognition for applications like meeting transcription, though it is incremental as it adapts existing models.
The paper tackles the challenge of recognizing overlapped speech in multi-lingual automatic speech recognition (ASR) models, proposing an adaptation approach that achieves effective transfer to a strong multilingual multi-talker ASR model with timestamp prediction, as demonstrated on the AMI and AliMeeting corpora.
State-of-the-art large-scale universal speech models (USMs) show a decent automatic speech recognition (ASR) performance across multiple domains and languages. However, it remains a challenge for these models to recognize overlapped speech, which is often seen in meeting conversations. We propose an approach to adapt USMs for multi-talker ASR. We first develop an enhanced version of serialized output training to jointly perform multi-talker ASR and utterance timestamp prediction. That is, we predict the ASR hypotheses for all speakers, count the speakers, and estimate the utterance timestamps at the same time. We further introduce a lightweight adapter module to maintain the multilingual property of the USMs even when we perform the adaptation with only a single language. Experimental results obtained using the AMI and AliMeeting corpora show that our proposed approach effectively transfers the USMs to a strong multilingual multi-talker ASR model with timestamp prediction capability.