Cross-Modal ASR Post-Processing System for Error Correction and Utterance Rejection
This addresses the problem of ASR errors degrading user experience and downstream tasks, representing an incremental improvement through multi-task learning and modality fusion.
The paper tackles errors in automatic speech recognition (ASR) by proposing a cross-modal post-processing system that fuses acoustic and textual features, achieving over 10% relative reduction in character error rate with about 1.7ms latency per token.
Although modern automatic speech recognition (ASR) systems can achieve high performance, they may produce errors that weaken readers' experience and do harm to downstream tasks. To improve the accuracy and reliability of ASR hypotheses, we propose a cross-modal post-processing system for speech recognizers, which 1) fuses acoustic features and textual features from different modalities, 2) joints a confidence estimator and an error corrector in multi-task learning fashion and 3) unifies error correction and utterance rejection modules. Compared with single-modal or single-task models, our proposed system is proved to be more effective and efficient. Experiment result shows that our post-processing system leads to more than 10% relative reduction of character error rate (CER) for both single-speaker and multi-speaker speech on our industrial ASR system, with about 1.7ms latency for each token, which ensures that extra latency introduced by post-processing is acceptable in streaming speech recognition.