Are Arabic Benchmarks Reliable? QIMMA's Quality-First Approach to LLM Evaluation
For Arabic NLP researchers and practitioners, QIMMA provides a more trustworthy evaluation framework, addressing the problem of unreliable benchmarks in a low-resource language setting.
QIMMA addresses the lack of reliable Arabic LLM evaluation by creating a quality-assured leaderboard that systematically validates benchmarks, resulting in a curated suite of over 52k samples. The approach improves evaluation reliability through automated and human review.
We present QIMMA, a quality-assured Arabic LLM leaderboard that places systematic benchmark validation at its core. Rather than aggregating existing resources as-is, QIMMA applies a multi-model assessment pipeline combining automated LLM judgment with human review to surface and resolve systematic quality issues in well-established Arabic benchmarks before evaluation. The result is a curated, multi-domain, multi-task evaluation suite of over 52k samples, grounded predominantly in native Arabic content; code evaluation tasks are the sole exception, as they are inherently language-agnostic. Transparent implementation via LightEval, EvalPlus and public release of per-sample inference outputs make QIMMA a reproducible and community-extensible foundation for Arabic NLP evaluation.