Palm: A Culturally Inclusive and Linguistically Diverse Dataset for Arabic LLMs
This addresses the problem of cultural insensitivity in AI for Arabic-speaking communities, though it is incremental as it builds on existing dataset efforts.
The authors tackled the lack of cultural and dialectal diversity in Arabic LLMs by creating a community-driven dataset covering all 22 Arab countries, and found that frontier LLMs have notable limitations, with closed-source models performing better but still flawed and smaller open-source models facing greater challenges.
As large language models (LLMs) become increasingly integrated into daily life, ensuring their cultural sensitivity and inclusivity is paramount. We introduce our dataset, a year-long community-driven project covering all 22 Arab countries. The dataset includes instructions (input, response pairs) in both Modern Standard Arabic (MSA) and dialectal Arabic (DA), spanning 20 diverse topics. Built by a team of 44 researchers across the Arab world, all of whom are authors of this paper, our dataset offers a broad, inclusive perspective. We use our dataset to evaluate the cultural and dialectal capabilities of several frontier LLMs, revealing notable limitations. For instance, while closed-source LLMs generally exhibit strong performance, they are not without flaws, and smaller open-source models face greater challenges. Moreover, certain countries (e.g., Egypt, the UAE) appear better represented than others (e.g., Iraq, Mauritania, Yemen). Our annotation guidelines, code, and data for reproducibility are publicly available.