Absher: A Benchmark for Evaluating Large Language Models Understanding of Saudi Dialects
This addresses the need for dialect-aware evaluation in Arabic NLP applications, particularly for linguistically diverse settings like Saudi Arabia, but it is incremental as it focuses on benchmarking rather than new methods.
The paper tackles the problem of evaluating large language models' understanding of Saudi dialects by introducing the Absher benchmark, which includes over 18,000 multiple-choice questions across six categories, and results show notable performance gaps, especially in cultural and contextual tasks.
As large language models (LLMs) become increasingly central to Arabic NLP applications, evaluating their understanding of regional dialects and cultural nuances is essential, particularly in linguistically diverse settings like Saudi Arabia. This paper introduces \texttt{Absher}, a comprehensive benchmark specifically designed to assess LLMs performance across major Saudi dialects. \texttt{Absher} comprises over 18,000 multiple-choice questions spanning six distinct categories: Meaning, True/False, Fill-in-the-Blank, Contextual Usage, Cultural Interpretation, and Location Recognition. These questions are derived from a curated dataset of dialectal words, phrases, and proverbs sourced from various regions of Saudi Arabia. We evaluate several state-of-the-art LLMs, including multilingual and Arabic-specific models. We also provide detailed insights into their capabilities and limitations. Our results reveal notable performance gaps, particularly in tasks requiring cultural inference or contextual understanding. Our findings highlight the urgent need for dialect-aware training and culturally aligned evaluation methodologies to improve LLMs performance in real-world Arabic applications.