JABER and SABER: Junior and Senior Arabic BERt
This addresses the need for better Arabic language models for natural language understanding tasks, though it is incremental as it builds on existing BERT architectures.
The authors tackled the problem of under-trained Arabic BERT models by introducing JABER and SABER, which achieved state-of-the-art performances on the ALUE benchmark and an NER benchmark.
Language-specific pre-trained models have proven to be more accurate than multilingual ones in a monolingual evaluation setting, Arabic is no exception. However, we found that previously released Arabic BERT models were significantly under-trained. In this technical report, we present JABER and SABER, Junior and Senior Arabic BERt respectively, our pre-trained language model prototypes dedicated for Arabic. We conduct an empirical study to systematically evaluate the performance of models across a diverse set of existing Arabic NLU tasks. Experimental results show that JABER and SABER achieve state-of-the-art performances on ALUE, a new benchmark for Arabic Language Understanding Evaluation, as well as on a well-established NER benchmark.