Cultural Value Differences of LLMs: Prompt, Language, and Model Size
This work addresses the problem of cultural bias in AI for users and developers, though it is incremental as it builds on existing research on LLM behavior.
The study investigated how large language models (LLMs) exhibit cultural values, finding that prompting language and model size significantly influence these values, with language causing divergent responses and larger model sizes having a greater impact than differences between models.
Our study aims to identify behavior patterns in cultural values exhibited by large language models (LLMs). The studied variants include question ordering, prompting language, and model size. Our experiments reveal that each tested LLM can efficiently behave with different cultural values. More interestingly: (i) LLMs exhibit relatively consistent cultural values when presented with prompts in a single language. (ii) The prompting language e.g., Chinese or English, can influence the expression of cultural values. The same question can elicit divergent cultural values when the same LLM is queried in a different language. (iii) Differences in sizes of the same model (e.g., Llama2-7B vs 13B vs 70B) have a more significant impact on their demonstrated cultural values than model differences (e.g., Llama2 vs Mixtral). Our experiments reveal that query language and model size of LLM are the main factors resulting in cultural value differences.