Compute Cost Amortized Transformer for Streaming ASR
This work addresses efficiency challenges in streaming ASR for real-time applications, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackles the problem of high computational cost in streaming automatic speech recognition by introducing a compute cost amortized Transformer architecture that dynamically creates sparse computation pathways at inference time, achieving a 60% reduction in compute cost with only a 3% relative increase in word error rate on LibriSpeech data.
We present a streaming, Transformer-based end-to-end automatic speech recognition (ASR) architecture which achieves efficient neural inference through compute cost amortization. Our architecture creates sparse computation pathways dynamically at inference time, resulting in selective use of compute resources throughout decoding, enabling significant reductions in compute with minimal impact on accuracy. The fully differentiable architecture is trained end-to-end with an accompanying lightweight arbitrator mechanism operating at the frame-level to make dynamic decisions on each input while a tunable loss function is used to regularize the overall level of compute against predictive performance. We report empirical results from experiments using the compute amortized Transformer-Transducer (T-T) model conducted on LibriSpeech data. Our best model can achieve a 60% compute cost reduction with only a 3% relative word error rate (WER) increase.