ALPS: A Diagnostic Challenge Set for Arabic Linguistic & Pragmatic Reasoning
This addresses the problem of shallow linguistic understanding in Arabic NLP for researchers and developers, though it is incremental as it builds on existing benchmark efforts by adding depth-focused diagnostics.
The authors tackled the lack of deep linguistic verification in Arabic NLP benchmarks by introducing ALPS, a native, expert-curated diagnostic challenge set, and found that models achieve high fluency but fail on fundamental morpho-syntactic dependencies, with error rates of 36.5% on diacritics-reliant tasks, while top commercial models like Gemini-3-flash reached 94.2% accuracy but Arabic-native models like Jais-2-70B only reached 83.6%.
While recent Arabic NLP benchmarks focus on scale, they often rely on synthetic or translated data which may benefit from deeper linguistic verification. We introduce ALPS (Arabic Linguistic & Pragmatic Suite), a native, expert-curated diagnostic challenge set probing Deep Semantics and Pragmatics, capabilities that complement specialized large-scale benchmarks. While broad-coverage benchmarks prioritize scale and multi-task coverage, ALPS targets the depth of linguistic understanding through 531 rigorously crafted questions across 15 tasks and 47 subtasks. We developed the dataset with deep expertise in Arabic linguistics, guaranteeing cultural authenticity and eliminating translation artifacts. Evaluating 23 diverse models (commercial, open-source, and Arabic-native) against a single-pass human performance (avg. 84.6% accuracy) and an expert-adjudicated oracle (99.2%), we reveal a critical dissociation: models achieve high fluency but fail on fundamental morpho-syntactic dependencies, with elevated error rates on morpho-syntactic dependencies (36.5% across diacritics-reliant tasks) compared to compositional semantics. While top commercial models (Gemini-3-flash at 94.2%) surpass the average single human, a substantial gap persists between commercial giants and Arabic-native models, with the best Arabic-specific model (Jais-2-70B at 83.6%) approaching but not matching human performance.