Improving Practical Aspects of End-to-End Multi-Talker Speech Recognition for Online and Offline Scenarios
This work addresses practical needs in real-time captioning and summarization for overlapping speech scenarios, representing an incremental advancement in end-to-end ASR systems.
The paper tackled the challenge of multi-talker speech recognition for both streaming and offline applications by extending Serialized Output Training with improvements like Continuous Speech Separation front-end and dual models, achieving enhanced accuracy and latency balance.
We extend the frameworks of Serialized Output Training (SOT) to address practical needs of both streaming and offline automatic speech recognition (ASR) applications. Our approach focuses on balancing latency and accuracy, catering to real-time captioning and summarization requirements. We propose several key improvements: (1) Leveraging Continuous Speech Separation (CSS) single-channel front-end with end-to-end (E2E) systems for highly overlapping scenarios, challenging the conventional wisdom of E2E versus cascaded setups. The CSS framework improves the accuracy of the ASR system by separating overlapped speech from multiple speakers. (2) Implementing dual models -- Conformer Transducer for streaming and Sequence-to-Sequence for offline -- or alternatively, a two-pass model based on cascaded encoders. (3) Exploring segment-based SOT (segSOT) which is better suited for offline scenarios while also enhancing readability of multi-talker transcriptions.