Streaming end-to-end multi-talker speech recognition
This work tackles the problem of real-time multi-talker speech recognition, which is crucial for applications like transcribing conversations and meetings, improving the usability of such systems for end-users.
This paper introduces the Streaming Unmixing and Recognition Transducer (SURT) for real-time multi-talker speech recognition, addressing the limitation of existing offline-only systems. The SURT model, particularly when trained with the Heuristic Error Assignment Training (HEAT) approach, achieves better accuracy than Permutation Invariant Training (PIT) and performs comparably to an offline baseline on the LibriSpeechMix dataset with a 150ms latency constraint.
End-to-end multi-talker speech recognition is an emerging research trend in the speech community due to its vast potential in applications such as conversation and meeting transcriptions. To the best of our knowledge, all existing research works are constrained in the offline scenario. In this work, we propose the Streaming Unmixing and Recognition Transducer (SURT) for end-to-end multi-talker speech recognition. Our model employs the Recurrent Neural Network Transducer (RNN-T) as the backbone that can meet various latency constraints. We study two different model architectures that are based on a speaker-differentiator encoder and a mask encoder respectively. To train this model, we investigate the widely used Permutation Invariant Training (PIT) approach and the Heuristic Error Assignment Training (HEAT) approach. Based on experiments on the publicly available LibriSpeechMix dataset, we show that HEAT can achieve better accuracy compared with PIT, and the SURT model with 150 milliseconds algorithmic latency constraint compares favorably with the offline sequence-to-sequence based baseline model in terms of accuracy.