An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognition
This work provides a comprehensive evaluation of pretrained representations for ASR researchers, though it is incremental as it builds on existing methods without introducing new paradigms.
The paper explored the application of self-supervised pretrained speech representations in end-to-end automatic speech recognition (E2E-ASR) across various benchmark corpora, finding that some representations, like HuBERT, achieved or outperformed state-of-the-art performance on datasets such as WSJ and WSJ0-2mix without model modifications.
Self-supervised pretraining on speech data has achieved a lot of progress. High-fidelity representation of the speech signal is learned from a lot of untranscribed data and shows promising performance. Recently, there are several works focusing on evaluating the quality of self-supervised pretrained representations on various tasks without domain restriction, e.g. SUPERB. However, such evaluations do not provide a comprehensive comparison among many ASR benchmark corpora. In this paper, we focus on the general applications of pretrained speech representations, on advanced end-to-end automatic speech recognition (E2E-ASR) models. We select several pretrained speech representations and present the experimental results on various open-source and publicly available corpora for E2E-ASR. Without any modification of the back-end model architectures or training strategy, some of the experiments with pretrained representations, e.g., WSJ, WSJ0-2mix with HuBERT, reach or outperform current state-of-the-art (SOTA) recognition performance. Moreover, we further explore more scenarios for whether the pretraining representations are effective, such as the cross-language or overlapped speech. The scripts, configuratons and the trained models have been released in ESPnet to let the community reproduce our experiments and improve them.