Tahakom LLM Guidelines and Recipes: From Pre-training Data to an Arabic LLM
This work addresses the problem of building effective Arabic LLMs for researchers and developers, but it is incremental as it builds on existing LLM techniques with domain-specific adaptations.
The paper tackles the challenges of developing large language models for Arabic by focusing on data curation, tokenizer design, and evaluation, proposing a corrective methodology for existing frameworks and sharing data and methods to advance Arabic language modeling.
Large Language Models (LLMs) have significantly advanced the field of natural language processing, enhancing capabilities in both language understanding and generation across diverse domains. However, developing LLMs for Arabic presents unique challenges. This paper explores these challenges by focusing on critical aspects such as data curation, tokenizer design, and evaluation. We detail our approach to the collection and filtration of Arabic pre-training datasets, assess the impact of various tokenizer designs on model performance, and examine the limitations of existing Arabic evaluation frameworks, for which we propose a systematic corrective methodology. To promote transparency and facilitate collaborative development, we share our data and methodologies, contributing to the advancement of language modeling, particularly for the Arabic language.