Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMs
This work addresses the problem of inaccurate cultural alignment assessments in LLMs for researchers and practitioners, advocating for more robust evaluation frameworks.
The paper challenges the use of closed-style multiple-choice surveys for evaluating cultural alignment in LLMs, showing that LLMs exhibit stronger alignment in unconstrained settings and that minor changes like reordering choices lead to inconsistent outputs.
A large number of studies rely on closed-style multiple-choice surveys to evaluate cultural alignment in Large Language Models (LLMs). In this work, we challenge this constrained evaluation paradigm and explore more realistic, unconstrained approaches. Using the World Values Survey (WVS) and Hofstede Cultural Dimensions as case studies, we demonstrate that LLMs exhibit stronger cultural alignment in less constrained settings, where responses are not forced. Additionally, we show that even minor changes, such as reordering survey choices, lead to inconsistent outputs, exposing the limitations of closed-style evaluations. Our findings advocate for more robust and flexible evaluation frameworks that focus on specific cultural proxies, encouraging more nuanced and accurate assessments of cultural alignment in LLMs.