Pretraining by Backtranslation for End-to-end ASR in Low-Resource Settings
This addresses the challenge of training ASR models with limited transcribed speech data, which is incremental as it builds on existing pretraining methods.
The paper tackled the problem of poor performance in attention-based encoder-decoder ASR models in low-resource settings by pretraining network parameters using text-based data and transcribed speech from other languages, resulting in a 20% average relative improvement over a baseline and a further 20-30% relative reduction in character error rate.
We explore training attention-based encoder-decoder ASR in low-resource settings. These models perform poorly when trained on small amounts of transcribed speech, in part because they depend on having sufficient target-side text to train the attention and decoder networks. In this paper we address this shortcoming by pretraining our network parameters using only text-based data and transcribed speech from other languages. We analyze the relative contributions of both sources of data. Across 3 test languages, our text-based approach resulted in a 20% average relative improvement over a text-based augmentation technique without pretraining. Using transcribed speech from nearby languages gives a further 20-30% relative reduction in character error rate.