MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmark
For NLP practitioners and AI safety researchers, this work highlights that multilingual models do not automatically achieve cultural alignment, revealing a critical gap that requires deeper intervention beyond current methods.
The paper introduces MSQA, a benchmark of 1,064 natively sourced questions across 11 languages and five cultural dimensions, and finds that LLMs exhibit substantial cultural degradation and a Locality Effect where cultural competence correlates with pre-training exposure rather than reasoning ability, with common inference-time fixes failing to resolve the issue.
Multilingual fluency often invites a stronger assumption: a model that can speak a user's language must also understand the culture encoded by that language. We call this the Illusion of Cultural Alignment. To test this assumption directly, we introduce MSQA, a benchmark of 1,064 natively sourced questions across 11 language groups, five cultural dimensions, and three difficulty tiers. Unlike translated benchmarks, MSQA targets locally grounded knowledge and reduces shortcuts from English-centric cross-lingual transfer. Evaluating 18 LLMs, we find substantial cultural degradation and a pronounced Locality Effect: cultural competence tracks pre-training exposure more closely than general reasoning ability. We further show that common inference-time remedies do not dissolve the illusion. Models remain overconfident on unfamiliar cultural questions, repeated sampling yields unstable rather than reliable correctness, and retrieval augmentation helps unevenly on long-tail facts. These findings indicate that cultural alignment cannot be inferred from multilingual ability alone and requires deeper intervention than calibration, sampling, or retrieval at inference time