Extrinsic Evaluation of Cultural Competence in Large Language Models
This work addresses the need for culturally relevant AI outputs for diverse users, but it is incremental as it builds on prior knowledge-based evaluations without introducing new methods.
The paper tackled the problem of evaluating cultural competence in large language models by focusing on extrinsic evaluation in text generation tasks like question answering and story generation, finding that model outputs vary with nationality cues but show weak correlations with cultural values.
Productive interactions between diverse users and language technologies require outputs from the latter to be culturally relevant and sensitive. Prior works have evaluated models' knowledge of cultural norms, values, and artifacts, without considering how this knowledge manifests in downstream applications. In this work, we focus on extrinsic evaluation of cultural competence in two text generation tasks, open-ended question answering and story generation. We quantitatively and qualitatively evaluate model outputs when an explicit cue of culture, specifically nationality, is perturbed in the prompts. Although we find that model outputs do vary when varying nationalities and feature culturally relevant words, we also find weak correlations between text similarity of outputs for different countries and the cultural values of these countries. Finally, we discuss important considerations in designing comprehensive evaluation of cultural competence in user-facing tasks.