Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languages
This work addresses the problem of adapting speech technologies to thousands of languages beyond those in pre-training, but it is incremental as it compares existing models without introducing new methods.
The study compared multilingual self-supervised (XLS-R) and weakly-supervised (Whisper) speech pre-training models for adaptation to unseen languages, finding that the number of hours per language and language family during pre-training predicts performance differences, with results based on fine-tuning on 13 unseen and 18 seen languages.
Recent models such as XLS-R and Whisper have made multilingual speech technologies more accessible by pre-training on audio from around 100 spoken languages each. However, there are thousands of spoken languages worldwide, and adapting to new languages is an important problem. In this work, we aim to understand which model adapts better to languages unseen during pre-training. We fine-tune both models on 13 unseen languages and 18 seen languages. Our results show that the number of hours seen per language and language family during pre-training is predictive of how the models compare, despite the significant differences in the pre-training methods.