AraMUS: Pushing the Limits of Data and Model Scale for Arabic Natural Language Processing
This work addresses the need for large-scale monolingual models in Arabic NLP, offering significant improvements for researchers and practitioners in this domain, though it is incremental in scaling existing methods.
The authors tackled the challenge of advancing Arabic natural language processing by developing AraMUS, an 11B-parameter pre-trained language model trained on 529GB of data, which achieved state-of-the-art performance on various tasks and demonstrated strong few-shot learning abilities.
Developing monolingual large Pre-trained Language Models (PLMs) is shown to be very successful in handling different tasks in Natural Language Processing (NLP). In this work, we present AraMUS, the largest Arabic PLM with 11B parameters trained on 529GB of high-quality Arabic textual data. AraMUS achieves state-of-the-art performances on a diverse set of Arabic classification and generative tasks. Moreover, AraMUS shows impressive few-shot learning abilities compared with the best existing Arabic PLMs.