DaKultur: Evaluating the Cultural Awareness of Language Models for Danish with Native Speakers
This addresses the problem of anglocentric and inappropriate responses for Danish speakers, though it is incremental as it focuses on a specific language and cultural evaluation.
The study tackled the lack of cultural awareness in large language models for underrepresented languages like Danish, finding that training on native-speaker data more than doubled response acceptance rates compared to using automatically translated data.
Large Language Models (LLMs) have seen widespread societal adoption. However, while they are able to interact with users in languages beyond English, they have been shown to lack cultural awareness, providing anglocentric or inappropriate responses for underrepresented language communities. To investigate this gap and disentangle linguistic versus cultural proficiency, we conduct the first cultural evaluation study for the mid-resource language of Danish, in which native speakers prompt different models to solve tasks requiring cultural awareness. Our analysis of the resulting 1,038 interactions from 63 demographically diverse participants highlights open challenges to cultural adaptation: Particularly, how currently employed automatically translated data are insufficient to train or measure cultural adaptation, and how training on native-speaker data can more than double response acceptance rates. We release our study data as DaKultur - the first native Danish cultural awareness dataset.