Transformers with convolutional context for ASR
This work addresses speech recognition for ASR systems, presenting an incremental improvement in transformer optimization and performance.
The paper tackled the problem of applying transformer networks to speech recognition by replacing sinusoidal positional embeddings with convolutionally learned input representations, achieving competitive word error rates of 4.7% and 12.9% on Librispeech test subsets without extra language model text.
The recent success of transformer networks for neural machine translation and other NLP tasks has led to a surge in research work trying to apply it for speech recognition. Recent efforts studied key research questions around ways of combining positional embedding with speech features, and stability of optimization for large scale learning of transformer networks. In this paper, we propose replacing the sinusoidal positional embedding for transformers with convolutionally learned input representations. These contextual representations provide subsequent transformer blocks with relative positional information needed for discovering long-range relationships between local concepts. The proposed system has favorable optimization characteristics where our reported results are produced with fixed learning rate of 1.0 and no warmup steps. The proposed model achieves a competitive 4.7% and 12.9% WER on the Librispeech ``test clean'' and ``test other'' subsets when no extra LM text is provided.