Single-Channel Multi-talker Speech Recognition with Permutation Invariant Training
This addresses the challenge of multi-talker speech recognition in real-world scenarios like meetings or crowded environments, representing a significant advance rather than an incremental improvement.
The paper tackles the problem of recognizing speech when multiple people talk simultaneously using only a single microphone, proposing architectures that extend permutation invariant training with feature separation and recognition modules. The result is a 45.0% relative reduction in word error rate for two-talker speech and 25.0% for three-talker speech compared to state-of-the-art single-talker systems.
Although great progresses have been made in automatic speech recognition (ASR), significant performance degradation is still observed when recognizing multi-talker mixed speech. In this paper, we propose and evaluate several architectures to address this problem under the assumption that only a single channel of mixed signal is available. Our technique extends permutation invariant training (PIT) by introducing the front-end feature separation module with the minimum mean square error (MSE) criterion and the back-end recognition module with the minimum cross entropy (CE) criterion. More specifically, during training we compute the average MSE or CE over the whole utterance for each possible utterance-level output-target assignment, pick the one with the minimum MSE or CE, and optimize for that assignment. This strategy elegantly solves the label permutation problem observed in the deep learning based multi-talker mixed speech separation and recognition systems. The proposed architectures are evaluated and compared on an artificially mixed AMI dataset with both two- and three-talker mixed speech. The experimental results indicate that our proposed architectures can cut the word error rate (WER) by 45.0% and 25.0% relatively against the state-of-the-art single-talker speech recognition system across all speakers when their energies are comparable, for two- and three-talker mixed speech, respectively. To our knowledge, this is the first work on the multi-talker mixed speech recognition on the challenging speaker-independent spontaneous large vocabulary continuous speech task.