MERaLiON-SpeechEncoder: Towards a Speech Foundation Model for Singapore and Beyond
It addresses speech processing needs in Singapore and the region, but is incremental as it builds on existing self-supervised learning methods.
The paper tackles the need for a speech foundation model tailored to Singapore and Southeast Asia by developing MERaLiON-SpeechEncoder, which shows improvements in spontaneous and Singapore speech recognition benchmarks while remaining competitive across ten other speech tasks.
This technical report describes the MERaLiON-SpeechEncoder, a foundation model designed to support a wide range of downstream speech applications. Developed as part of Singapore's National Multimodal Large Language Model Programme, the MERaLiON-SpeechEncoder is tailored to address the speech processing needs in Singapore and the surrounding Southeast Asian region. The model currently supports mainly English, including the variety spoken in Singapore. We are actively expanding our datasets to gradually cover other languages in subsequent releases. The MERaLiON-SpeechEncoder was pre-trained from scratch on 200,000 hours of unlabelled speech data using a self-supervised learning approach based on masked language modelling. We describe our training procedure and hyperparameter tuning experiments in detail below. Our evaluation demonstrates improvements to spontaneous and Singapore speech benchmarks for speech recognition, while remaining competitive to other state-of-the-art speech encoders across ten other speech tasks. We commit to releasing our model, supporting broader research endeavours, both in Singapore and beyond.