AmharicStoryQA: A Multicultural Story Question Answering Benchmark in Amharic
This addresses the need for culturally grounded benchmarks to improve narrative understanding in low-resource languages like Amharic, though it is incremental as it builds on existing multilingual evaluation efforts.
The authors tackled the problem of overlooking cultural variation within a single language in multilingual LLM evaluations by introducing AmharicStoryQA, a benchmark based on culturally diverse Amharic narratives, revealing significant gaps in narrative understanding and regional differences in LLM performance.
With the growing emphasis on multilingual and cultural evaluation benchmarks for large language models, language and culture are often treated as synonymous, and performance is commonly used as a proxy for a models understanding of a given language. In this work, we argue that such evaluations overlook meaningful cultural variation that exists within a single language. We address this gap by focusing on narratives from different regions of Ethiopia and demonstrate that, despite shared linguistic characteristics, region-specific and domain-specific content substantially influences language evaluation outcomes. To this end, we introduce \textbf{\textit{AmharicStoryQA}}, a long-sequence story question answering benchmark grounded in culturally diverse narratives from Amharic-speaking regions. Using this benchmark, we reveal a significant narrative understanding gap in existing LLMs, highlight pronounced regional differences in evaluation results, and show that supervised fine-tuning yields uneven improvements across regions and evaluation settings. Our findings emphasize the need for culturally grounded benchmarks that go beyond language-level evaluation to more accurately assess and improve narrative understanding in low-resource languages.