Deploying self-supervised learning in the wild for hybrid automatic speech recognition
This work addresses the challenge of applying SSL to real-world, untranscribed audio for ASR, though it is incremental as it builds on existing SSL methods.
The paper tackled the problem of deploying self-supervised learning (SSL) for hybrid automatic speech recognition (ASR) using uncurated audio data, showing that SSL pre-training with in-domain uncurated data achieves better performance compared to out-domain strategies.
Self-supervised learning (SSL) methods have proven to be very successful in automatic speech recognition (ASR). These great improvements have been reported mostly based on highly curated datasets such as LibriSpeech for non-streaming End-to-End ASR models. However, the pivotal characteristics of SSL is to be utilized for any untranscribed audio data. In this paper, we provide a full exploration on how to utilize uncurated audio data in SSL from data pre-processing to deploying an streaming hybrid ASR model. More specifically, we present (1) the effect of Audio Event Detection (AED) model in data pre-processing pipeline (2) analysis on choosing optimizer and learning rate scheduling (3) comparison of recently developed contrastive losses, (4) comparison of various pre-training strategies such as utilization of in-domain versus out-domain pre-training data, monolingual versus multilingual pre-training data, multi-head multilingual SSL versus single-head multilingual SSL and supervised pre-training versus SSL. The experimental results show that SSL pre-training with in-domain uncurated data can achieve better performance in comparison to all the alternative out-domain pre-training strategies.