BA-SOT: Boundary-Aware Serialized Output Training for Multi-Talker ASR
This work addresses the challenge of accurately transcribing overlapping speech in multi-talker ASR, which is incremental as it builds upon existing serialized output training methods.
The paper tackles the problem of frequent speaker changes making speaker change prediction difficult in multi-talker automatic speech recognition (ASR) by proposing boundary-aware serialized output training (BA-SOT), which reduces character error rate (CER) by 5.1% and utterance-dependent CER (UD-CER) by 14.0% compared to original SOT, with further reductions of 8.4% and 19.9% when using pre-trained model initialization.
The recently proposed serialized output training (SOT) simplifies multi-talker automatic speech recognition (ASR) by generating speaker transcriptions separated by a special token. However, frequent speaker changes can make speaker change prediction difficult. To address this, we propose boundary-aware serialized output training (BA-SOT), which explicitly incorporates boundary knowledge into the decoder via a speaker change detection task and boundary constraint loss. We also introduce a two-stage connectionist temporal classification (CTC) strategy that incorporates token-level SOT CTC to restore temporal context information. Besides typical character error rate (CER), we introduce utterance-dependent character error rate (UD-CER) to further measure the precision of speaker change prediction. Compared to original SOT, BA-SOT reduces CER/UD-CER by 5.1%/14.0%, and leveraging a pre-trained ASR model for BA-SOT model initialization further reduces CER/UD-CER by 8.4%/19.9%.