FSR: Accelerating the Inference Process of Transducer-Based Models by Applying Fast-Skip Regularization
This work addresses inference speed bottlenecks for speech recognition systems using transducer models, offering a practical improvement for real-time applications.
The paper tackles the inefficiency of transducer-based models in speech recognition by introducing fast-skip regularization, which aligns blank token predictions with a CTC model to skip them during inference, resulting in a nearly 4x speedup with minimal performance degradation on the AISHELL-1 dataset.
Transducer-based models, such as RNN-Transducer and transformer-transducer, have achieved great success in speech recognition. A typical transducer model decodes the output sequence conditioned on the current acoustic state and previously predicted tokens step by step. Statistically, The number of blank tokens in the prediction results accounts for nearly 90\% of all tokens. It takes a lot of computation and time to predict the blank tokens, but only the non-blank tokens will appear in the final output sequence. Therefore, we propose a method named fast-skip regularization, which tries to align the blank position predicted by a transducer with that predicted by a CTC model. During the inference, the transducer model can predict the blank tokens in advance by a simple CTC project layer without many complicated forward calculations of the transducer decoder and then skip them, which will reduce the computation and improve the inference speed greatly. All experiments are conducted on a public Chinese mandarin dataset AISHELL-1. The results show that the fast-skip regularization can indeed help the transducer model learn the blank position alignments. Besides, the inference with fast-skip can be speeded up nearly 4 times with only a little performance degradation.