Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook
This addresses the problem of aligning LLMs with diverse cultural values for safety and engagement, offering a more nuanced evaluation method than existing benchmarks.
The paper tackled the challenge of evaluating cultural value alignment in LLMs by introducing DOVE, a distributional evaluation framework that compares human-written and LLM-generated text distributions, achieving a 31.56% correlation with downstream tasks.
As LLMs are globally deployed, aligning their cultural value orientations is critical for safety and user engagement. However, existing benchmarks face the Construct-Composition-Context ($C^3$) challenge: relying on discriminative, multiple-choice formats that probe value knowledge rather than true orientations, overlook subcultural heterogeneity, and mismatch with real-world open-ended generation. We introduce DOVE, a distributional evaluation framework that directly compares human-written text distributions with LLM-generated outputs. DOVE utilizes a rate-distortion variational optimization objective to construct a compact value-codebook from 10K documents, mapping text into a structured value space to filter semantic noise. Alignment is measured using unbalanced optimal transport, capturing intra-cultural distributional structures and sub-group diversity. Experiments across 12 LLMs show that DOVE achieves superior predictive validity, attaining a 31.56% correlation with downstream tasks, while maintaining high reliability with as few as 500 samples per culture.