Accelerating Transducers through Adjacent Token Merging
This addresses computational bottlenecks in ASR systems, particularly for long speech signals, but is incremental as it builds on existing token merging techniques.
The paper tackles the inefficiency of Transformer-based acoustic encoders in automatic speech recognition by proposing Adjacent Token Merging (A-ToMe), which reduces tokens by 57% and improves GPU inference speed by 70% without significant accuracy loss on LibriSpeech.
Recent end-to-end automatic speech recognition (ASR) systems often utilize a Transformer-based acoustic encoder that generates embedding at a high frame rate. However, this design is inefficient, particularly for long speech signals due to the quadratic computation of self-attention. To address this, we propose a new method, Adjacent Token Merging (A-ToMe), which gradually combines adjacent tokens with high similarity scores between their key values. In this way, the total time step could be reduced, and the inference of both the encoder and joint network is accelerated. Experiments on LibriSpeech show that our method can reduce 57% of tokens and improve the inference speed on GPU by 70% without any notable loss of accuracy. Additionally, we demonstrate that A-ToMe is also an effective solution to reduce tokens in long-form ASR, where the input speech consists of multiple utterances.