Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages
This addresses the challenge of building efficient and high-performing ASR systems for a broad range of languages, though it is incremental in scaling existing methods.
The paper tackles the problem of scaling automatic speech recognition to over 100 languages by introducing the Universal Speech Model (USM), which achieves state-of-the-art performance on multilingual ASR and speech-to-text translation tasks, with comparable or better results than Whisper using only 1/7-th the labeled data.
We introduce the Universal Speech Model (USM), a single large model that performs automatic speech recognition (ASR) across 100+ languages. This is achieved by pre-training the encoder of the model on a large unlabeled multilingual dataset of 12 million (M) hours spanning over 300 languages, and fine-tuning on a smaller labeled dataset. We use multilingual pre-training with random-projection quantization and speech-text modality matching to achieve state-of-the-art performance on downstream multilingual ASR and speech-to-text translation tasks. We also demonstrate that despite using a labeled training set 1/7-th the size of that used for the Whisper model, our model exhibits comparable or better performance on both in-domain and out-of-domain speech recognition tasks across many languages.