Beyond MCQ: An Open-Ended Arabic Cultural QA Benchmark with Dialect Variants
This addresses the need for culturally and linguistically inclusive evaluation in AI, particularly for Arabic dialects, though it is incremental as it builds on existing datasets and methods.
The paper tackled the problem of uneven performance of Large Language Models on culturally grounded and dialectal content in Arabic by creating an open-ended QA benchmark with dialect variants, finding that models underperform on Arabic dialects and struggle with open-ended questions despite improvements from chain-of-thought fine-tuning.
Large Language Models (LLMs) are increasingly used to answer everyday questions, yet their performance on culturally grounded and dialectal content remains uneven across languages. We propose a comprehensive method that (i) translates Modern Standard Arabic (MSA) multiple-choice questions (MCQs) into English and several Arabic dialects, (ii) converts them into open-ended questions (OEQs), (iii) benchmarks a range of zero-shot and fine-tuned LLMs under both MCQ and OEQ settings, and (iv) generates chain-of-thought (CoT) rationales to fine-tune models for step-by-step reasoning. Using this method, we extend an existing dataset in which QAs are parallelly aligned across multiple language varieties, making it, to our knowledge, the first of its kind. We conduct extensive experiments with both open and closed models. Our findings show that (i) models underperform on Arabic dialects, revealing persistent gaps in culturally grounded and dialect-specific knowledge; (ii) Arabic-centric models perform well on MCQs but struggle with OEQs; and (iii) CoT improves judged correctness while yielding mixed n-gram-based metrics. The developed dataset will be publicly released to support further research on culturally and linguistically inclusive evaluation.