The Arabic Generality Score: Another Dimension of Modeling Arabic Dialectness
This provides a scalable, linguistically grounded method for NLP researchers working on Arabic dialects, though it is incremental as it complements existing measures like ALDi.
The authors tackled the problem of modeling Arabic dialect variation as a continuous variable by proposing the Arabic Generality Score (AGS) to quantify word usage across dialects, and their regression model outperformed state-of-the-art dialect ID systems on a multi-dialect benchmark.
Arabic dialects form a diverse continuum, yet NLP models often treat them as discrete categories. Recent work addresses this issue by modeling dialectness as a continuous variable, notably through the Arabic Level of Dialectness (ALDi). However, ALDi reduces complex variation to a single dimension. We propose a complementary measure: the Arabic Generality Score (AGS), which quantifies how widely a word is used across dialects. We introduce a pipeline that combines word alignment, etymology-aware edit distance, and smoothing to annotate a parallel corpus with word-level AGS. A regression model is then trained to predict AGS in context. Our approach outperforms strong baselines, including state-of-the-art dialect ID systems, on a multi-dialect benchmark. AGS offers a scalable, linguistically grounded way to model lexical generality, enriching representations of Arabic dialectness.