CultureLLM: Incorporating Cultural Differences into Large Language Models
This addresses cultural bias in LLMs, which affects users from diverse cultural backgrounds, by providing a more inclusive and efficient solution, though it is incremental as it builds on existing fine-tuning and data augmentation techniques.
The paper tackles the problem of cultural bias in large language models (LLMs) due to training data dominance from English corpora, proposing CultureLLM, a cost-effective method that uses World Value Survey seed data and semantic data augmentation to fine-tune models for 9 cultures, achieving significant performance gains over GPT-3.5 (by 8.1%) and Gemini Pro (by 9.5%) with comparable or better results than GPT-4 on 60 culture-related datasets.
Large language models (LLMs) are reported to be partial to certain cultures owing to the training data dominance from the English corpora. Since multilingual cultural data are often expensive to collect, existing efforts handle this by prompt engineering or culture-specific pre-training. However, they might overlook the knowledge deficiency of low-resource culture and require extensive computing resources. In this paper, we propose CultureLLM, a cost-effective solution to incorporate cultural differences into LLMs. CultureLLM adopts World Value Survey (WVS) as seed data and generates semantically equivalent training data via the proposed semantic data augmentation. Using only 50 seed samples from WVS with augmented data, we fine-tune culture-specific LLMs and one unified model (CultureLLM-One) for 9 cultures covering rich and low-resource languages. Extensive experiments on 60 culture-related datasets demonstrate that CultureLLM significantly outperforms various counterparts such as GPT-3.5 (by 8.1%) and Gemini Pro (by 9.5%) with comparable performance to GPT-4 or even better. Our human study shows that the generated samples are semantically equivalent to the original samples, providing an effective solution for LLMs augmentation. Code is released at https://github.com/Scarelette/CultureLLM.