Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Compute
This work reduces barriers to self-supervised learning for researchers with limited compute resources, making it more accessible and reproducible, though it is incremental as it builds on existing HuBERT methods.
The authors tackled the problem of high computational requirements for self-supervised learning in speech processing by optimizing HuBERT pre-training to fit academic constraints, achieving similar performance with only 8 GPUs instead of 32 or 128 and improving over HuBERT on several tasks within one iteration.
Self-supervised learning (SSL) has led to great strides in speech processing. However, the resources needed to train these models has become prohibitively large as they continue to scale. Currently, only a few groups with substantial resources are capable of creating SSL models, which harms reproducibility. In this work, we optimize HuBERT SSL to fit in academic constraints. We reproduce HuBERT independently from the original implementation, with no performance loss. Our code and training optimizations make SSL feasible with only 8 GPUs, instead of the 32 used in the original work. We also explore a semi-supervised route, using an ASR model to skip the first pre-training iteration. Within one iteration of pre-training, our models improve over HuBERT on several tasks. Furthermore, our HuBERT Large variant requires only 8 GPUs, achieving similar performance to the original trained on 128. As our contribution to the community, all models, configurations, and code are made open-source in ESPnet.