Mixture Encoder for Joint Speech Separation and Recognition
This work addresses multi-speaker ASR for real-world applications, presenting an incremental improvement over existing modular and end-to-end methods.
The paper tackles the problem of multi-speaker automatic speech recognition by proposing a middle-ground approach that combines explicit speech separation with mixture speech information to reduce error propagation, achieving a 7% relative improvement in word error rate on the SMS-WSJ task.
Multi-speaker automatic speech recognition (ASR) is crucial for many real-world applications, but it requires dedicated modeling techniques. Existing approaches can be divided into modular and end-to-end methods. Modular approaches separate speakers and recognize each of them with a single-speaker ASR system. End-to-end models process overlapped speech directly in a single, powerful neural network. This work proposes a middle-ground approach that leverages explicit speech separation similarly to the modular approach but also incorporates mixture speech information directly into the ASR module in order to mitigate the propagation of errors made by the speech separator. We also explore a way to exchange cross-speaker context information through a layer that combines information of the individual speakers. Our system is optimized through separate and joint training stages and achieves a relative improvement of 7% in word error rate over a purely modular setup on the SMS-WSJ task.