DiariST: Streaming Speech Translation with Speaker Diarization
This work addresses the challenge of real-time speech translation with speaker identification in multi-talker conversations, which is incremental as it builds on existing methods for multi-talker speech recognition.
The authors tackled the problem of streaming speech translation with speaker diarization for conversation recordings, developing DiariST, which achieves strong performance compared to offline systems like Whisper while handling overlapping speech in real-time.
End-to-end speech translation (ST) for conversation recordings involves several under-explored challenges such as speaker diarization (SD) without accurate word time stamps and handling of overlapping speech in a streaming fashion. In this work, we propose DiariST, the first streaming ST and SD solution. It is built upon a neural transducer-based streaming ST system and integrates token-level serialized output training and t-vector, which were originally developed for multi-talker speech recognition. Due to the absence of evaluation benchmarks in this area, we develop a new evaluation dataset, DiariST-AliMeeting, by translating the reference Chinese transcriptions of the AliMeeting corpus into English. We also propose new metrics, called speaker-agnostic BLEU and speaker-attributed BLEU, to measure the ST quality while taking SD accuracy into account. Our system achieves a strong ST and SD capability compared to offline systems based on Whisper, while performing streaming inference for overlapping speech. To facilitate the research in this new direction, we release the evaluation data, the offline baseline systems, and the evaluation code.