BALSAM: A Platform for Benchmarking Arabic Large Language Models
This addresses the problem of limited and low-quality benchmarks for Arabic LLMs, which hinders development and evaluation in the Arabic NLP community, though it is incremental as it builds on existing benchmarking concepts.
The paper tackles the lag in Arabic large language model performance by introducing BALSAM, a comprehensive benchmark with 78 NLP tasks and 52K examples, providing a centralized platform for blind evaluation to measure progress and mitigate data contamination.
The impressive advancement of Large Language Models (LLMs) in English has not been matched across all languages. In particular, LLM performance in Arabic lags behind, due to data scarcity, linguistic diversity of Arabic and its dialects, morphological complexity, etc. Progress is further hindered by the quality of Arabic benchmarks, which typically rely on static, publicly available data, lack comprehensive task coverage, or do not provide dedicated platforms with blind test sets. This makes it challenging to measure actual progress and to mitigate data contamination. Here, we aim to bridge these gaps. In particular, we introduce BALSAM, a comprehensive, community-driven benchmark aimed at advancing Arabic LLM development and evaluation. It includes 78 NLP tasks from 14 broad categories, with 52K examples divided into 37K test and 15K development, and a centralized, transparent platform for blind evaluation. We envision BALSAM as a unifying platform that sets standards and promotes collaborative research to advance Arabic LLM capabilities.