Multi-Encoder Learning and Stream Fusion for Transformer-Based End-to-End Automatic Speech Recognition
This work addresses the challenge of enhancing speech recognition accuracy for users in noisy or varied acoustic environments, representing an incremental improvement over existing transformer methods.
The paper tackles the problem of improving transformer-based end-to-end automatic speech recognition by exploring stream fusion techniques, achieving a 19% relative reduction in word error rate on the Wall Street Journal dataset compared to the benchmark.
Stream fusion, also known as system combination, is a common technique in automatic speech recognition for traditional hybrid hidden Markov model approaches, yet mostly unexplored for modern deep neural network end-to-end model architectures. Here, we investigate various fusion techniques for the all-attention-based encoder-decoder architecture known as the transformer, striving to achieve optimal fusion by investigating different fusion levels in an example single-microphone setting with fusion of standard magnitude and phase features. We introduce a novel multi-encoder learning method that performs a weighted combination of two encoder-decoder multi-head attention outputs only during training. Employing then only the magnitude feature encoder in inference, we are able to show consistent improvement on Wall Street Journal (WSJ) with language model and on Librispeech, without increase in runtime or parameters. Combining two such multi-encoder trained models by a simple late fusion in inference, we achieve state-of-the-art performance for transformer-based models on WSJ with a significant WER reduction of 19% relative compared to the current benchmark approach.