Continuous Streaming Multi-Talker ASR with Dual-path Transducers
This addresses the challenge of real-time ASR for multi-speaker conversations, such as in meetings, but is incremental as it adapts existing dual-path strategies from speech separation to this specific domain.
The paper tackled the problem of streaming multi-talker automatic speech recognition (ASR) in multi-turn meetings, showing that naive extensions of single-turn models degrade performance, and proposed dual-path models that improve word error rate (WER) and convergence speed, achieving competitive results with offline methods on LibriCSS data.
Streaming recognition of multi-talker conversations has so far been evaluated only for 2-speaker single-turn sessions. In this paper, we investigate it for multi-turn meetings containing multiple speakers using the Streaming Unmixing and Recognition Transducer (SURT) model, and show that naively extending the single-turn model to this harder setting incurs a performance penalty. As a solution, we propose the dual-path (DP) modeling strategy first used for time-domain speech separation. We experiment with LSTM and Transformer based DP models, and show that they improve word error rate (WER) performance while yielding faster convergence. We also explore training strategies such as chunk width randomization and curriculum learning for these models, and demonstrate their importance through ablation studies. Finally, we evaluate our models on the LibriCSS meeting data, where they perform competitively with offline separation-based methods.