MSA-ASR: Efficient Multilingual Speaker Attribution with frozen ASR Models
This work addresses the challenge of efficiently transcribing and attributing speech to speakers in multilingual settings, offering a more adaptable and general solution compared to existing complex or fine-tuning-heavy methods.
The paper tackles the problem of speaker-attributed automatic speech recognition (SA-ASR) by introducing a method that uses a frozen multilingual ASR model to incorporate speaker attribution into transcriptions, achieving competitive performance on diverse multilingual datasets, including those with overlapping speech.
Speaker-attributed automatic speech recognition (SA-ASR) aims to transcribe speech while assigning transcripts to the corresponding speakers accurately. Existing methods often rely on complex modular systems or require extensive fine-tuning of joint modules, limiting their adaptability and general efficiency. This paper introduces a novel approach, leveraging a frozen multilingual ASR model to incorporate speaker attribution into the transcriptions, using only standard monolingual ASR datasets. Our method involves training a speaker module to predict speaker embeddings based on weak labels without requiring additional ASR model modifications. Despite being trained exclusively with non-overlapping monolingual data, our approach effectively extracts speaker attributes across diverse multilingual datasets, including those with overlapping speech. Experimental results demonstrate competitive performance compared to strong baselines, highlighting the model's robustness and potential for practical applications.