From Guidelines to Practice: A New Paradigm for Arabic Language Model Evaluation
It addresses evaluation issues for Arabic language models, emphasizing cultural competence, but is incremental as it builds on existing evaluation practices with a new dataset.
This paper tackled gaps in Arabic language model evaluation by establishing guidelines and introducing the Arabic Depth Mini Dataset (ADMD) with 490 questions across ten domains, evaluating five leading models and finding Claude 3.5 Sonnet had the highest overall accuracy at 30%.
This paper addresses critical gaps in Arabic language model evaluation by establishing comprehensive theoretical guidelines and introducing a novel evaluation framework. We first analyze existing Arabic evaluation datasets, identifying significant issues in linguistic accuracy, cultural alignment, and methodological rigor. To address these limitations in LLMs, we present the Arabic Depth Mini Dataset (ADMD), a carefully curated collection of 490 challenging questions spanning ten major domains (42 sub-domains, see Figure 1. Using ADMD, we evaluate five leading language models: GPT-4, Claude 3.5 Sonnet, Gemini Flash 1.5, CommandR 100B, and Qwen-Max. Our results reveal significant variations in model performance across different domains, with particular challenges in areas requiring deep cultural understanding and specialized knowledge. Claude 3.5 Sonnet demonstrated the highest overall accuracy at 30\%, showing relative strength in mathematical theory in Arabic, Arabic language, and islamic domains. This work provides both theoretical foundations and practical insights for improving Arabic language model evaluation, emphasizing the importance of cultural competence alongside technical capabilities.