AraModernBERT: Transtokenized Initialization and Long-Context Encoder Modeling for Arabic
This work addresses the gap in modern encoder architectures for Arabic, providing practical adaptations for Arabic-derived scripts, though it is incremental as it adapts an existing method to a new language.
The authors tackled the adaptation of ModernBERT encoder architecture to Arabic, focusing on transtokenized embedding initialization and long-context modeling up to 8,192 tokens, resulting in dramatic improvements in masked language modeling and strong performance on downstream tasks like inference and named entity recognition.
Encoder-only transformer models remain widely used for discriminative NLP tasks, yet recent architectural advances have largely focused on English. In this work, we present AraModernBERT, an adaptation of the ModernBERT encoder architecture to Arabic, and study the impact of transtokenized embedding initialization and native long-context modeling up to 8,192 tokens. We show that transtokenization is essential for Arabic language modeling, yielding dramatic improvements in masked language modeling performance compared to non-transtokenized initialization. We further demonstrate that AraModernBERT supports stable and effective long-context modeling, achieving improved intrinsic language modeling performance at extended sequence lengths. Downstream evaluations on Arabic natural language understanding tasks, including inference, offensive language detection, question-question similarity, and named entity recognition, confirm strong transfer to discriminative and sequence labeling settings. Our results highlight practical considerations for adapting modern encoder architectures to Arabic and other languages written in Arabic-derived scripts.