Enhancing Speech Recognition Decoding via Layer Aggregation
This work addresses a specific bottleneck in speech recognition decoding for researchers and practitioners, offering an incremental improvement over existing methods.
The paper tackled the problem of suboptimal beam search decoding in speech recognition by showing that top-layer predictions can be too confident and local, and proposed aggregating multiple layers to improve performance, achieving up to 10% WER and 22% CER reductions on Librispeech.
Recently proposed speech recognition systems are designed to predict using representations generated by their top layers, employing greedy decoding which isolates each timestep from the rest of the sequence. Aiming for improved performance, a beam search algorithm is frequently utilized and a language model is incorporated to assist with ranking the top candidates. In this work, we experiment with several speech recognition models and find that logits predicted using the top layers may hamper beam search from achieving optimal results. Specifically, we show that fined-tuned Wav2Vec 2.0 and HuBERT yield highly confident predictions, and hypothesize that the predictions are based on local information and may not take full advantage of the information encoded in intermediate layers. To this end, we perform a layer analysis to reveal and visualize how predictions evolve throughout the inference flow. We then propose a prediction method that aggregates the top M layers, potentially leveraging useful information encoded in intermediate layers and relaxing model confidence. We showcase the effectiveness of our approach via beam search decoding, conducting our experiments on Librispeech test and dev sets and achieving WER, and CER reduction of up to 10% and 22%, respectively.