A Token-Wise Beam Search Algorithm for RNN-T
This work addresses decoding speed bottlenecks in speech recognition systems, offering incremental improvements for practical applications.
The paper tackles the inefficiency of standard RNN-T decoding algorithms by introducing a token-wise beam search that batches joint network calls across time segments, resulting in 20%-96% decoding speedups and up to 11% relative improvement in oracle word error rate.
Standard Recurrent Neural Network Transducers (RNN-T) decoding algorithms for speech recognition are iterating over the time axis, such that one time step is decoded before moving on to the next time step. Those algorithms result in a large number of calls to the joint network, which were shown in previous work to be an important factor that reduces decoding speed. We present a decoding beam search algorithm that batches the joint network calls across a segment of time steps, which results in 20%-96% decoding speedups consistently across all models and settings experimented with. In addition, aggregating emission probabilities over a segment may be seen as a better approximation to finding the most likely model output, causing our algorithm to improve oracle word error rate by up to 11% relative as the segment size increases, and to slightly improve general word error rate.