VarArray Meets t-SOT: Advancing the State of the Art of Streaming Distant Conversational Speech Recognition
This work addresses the challenge of real-time transcription in multi-talker conversational settings like meetings, representing an incremental improvement by combining existing technologies.
The paper tackles the problem of streaming automatic speech recognition for overlapping speech from distant microphone arrays with arbitrary geometry, achieving state-of-the-art word error rates of 13.7% and 15.5% on AMI development and evaluation sets while maintaining streaming inference.
This paper presents a novel streaming automatic speech recognition (ASR) framework for multi-talker overlapping speech captured by a distant microphone array with an arbitrary geometry. Our framework, named t-SOT-VA, capitalizes on independently developed two recent technologies; array-geometry-agnostic continuous speech separation, or VarArray, and streaming multi-talker ASR based on token-level serialized output training (t-SOT). To combine the best of both technologies, we newly design a t-SOT-based ASR model that generates a serialized multi-talker transcription based on two separated speech signals from VarArray. We also propose a pre-training scheme for such an ASR model where we simulate VarArray's output signals based on monaural single-talker ASR training data. Conversation transcription experiments using the AMI meeting corpus show that the system based on the proposed framework significantly outperforms conventional ones. Our system achieves the state-of-the-art word error rates of 13.7% and 15.5% for the AMI development and evaluation sets, respectively, in the multiple-distant-microphone setting while retaining the streaming inference capability.