End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures
This work addresses performance gaps in ASR for speech recognition applications, but it is incremental as it builds on existing pseudo-labeling and architecture methods.
The paper tackled improving end-to-end automatic speech recognition (ASR) by applying pseudo-labeling for semi-supervised training across modern architectures like ResNet, ConvNets, and Transformers, achieving new state-of-the-art results in both supervised and semi-supervised settings on the LibriSpeech dataset.
We study pseudo-labeling for the semi-supervised training of ResNet, Time-Depth Separable ConvNets, and Transformers for speech recognition, with either CTC or Seq2Seq loss functions. We perform experiments on the standard LibriSpeech dataset, and leverage additional unlabeled data from LibriVox through pseudo-labeling. We show that while Transformer-based acoustic models have superior performance with the supervised dataset alone, semi-supervision improves all models across architectures and loss functions and bridges much of the performance gaps between them. In doing so, we reach a new state-of-the-art for end-to-end acoustic models decoded with an external language model in the standard supervised learning setting, and a new absolute state-of-the-art with semi-supervised training. Finally, we study the effect of leveraging different amounts of unlabeled audio, propose several ways of evaluating the characteristics of unlabeled audio which improve acoustic modeling, and show that acoustic models trained with more audio rely less on external language models.