A Unified Speech LLM for Diarization and Speech Recognition in Multilingual Conversations
This addresses the challenge of processing natural multilingual conversations for speech technology applications, though it is incremental as it builds on existing Speech LLM paradigms.
The paper tackled the problem of limited effectiveness of Speech LLMs in real-world multilingual conversations by proposing a unified model for joint diarization and speech recognition, achieving a 54.87% relative improvement in tcpWER/tcpCER over the baseline.
Speech Large Language Models (Speech LLMs) have emerged as a crucial paradigm in recent years, extending the capabilities of traditional LLMs to speech tasks such as automatic speech recognition (ASR) and spoken dialogue modeling. However, their effectiveness in real-world multilingual conversations remains limited by the scarcity of data that captures natural conversational phenomena. To address this, the MLC-SLM Challenge provides a multilingual conversational dataset and evaluates models on two tasks: ASR with oracle segmentation (Task I) and joint diarization and recognition without oracle information (Task II). In this paper, we focus on Task II and propose a unified speech LLM that jointly performs diarization and ASR in an end-to-end manner. By reformulating the training data format and modifying the inference procedure, our model addresses the ambiguity inherent in pre-segmented audio and achieves a 54.87\% relative improvement in tcpWER/tcpCER over the baseline, ranking 8th overall, despite using a smaller LLM backbone. We also report results from Task I using a fine-tuned speech LLM.