Evaluating and Improving Cultural Awareness of Reward Models for LLM Alignment
This work addresses the challenge of aligning LLMs with diverse cultures, which is crucial for global deployment, though it is incremental as it builds on existing reward modeling frameworks.
The paper tackled the problem of evaluating and improving the cultural awareness of reward models for LLM alignment by introducing the CARB benchmark covering 10 cultures across 4 domains, revealing deficiencies in existing models and showing a positive correlation with downstream tasks. It proposed Think-as-Locals, which uses RLVR to mitigate spurious correlations and improve cultural reasoning, validated by experimental results.
Reward models (RMs) are crucial for aligning large language models (LLMs) with diverse cultures. Consequently, evaluating their cultural awareness is essential for further advancing global alignment of LLMs. However, existing RM evaluations fall short in assessing cultural awareness due to the scarcity of culturally relevant evaluation datasets. To fill this gap, we propose Cultural Awareness Reward modeling Benchmark (CARB), covering 10 distinct cultures across 4 cultural domains. Our extensive evaluation of state-of-the-art RMs reveals their deficiencies in modeling cultural awareness and demonstrates a positive correlation between performance on CARB and downstream multilingual cultural alignment tasks. Further analysis identifies the spurious correlations within culture-aware reward modeling, wherein RM's scoring relies predominantly on surface-level features rather than authentic cultural nuance understanding. To address these, we propose Think-as-Locals to elicit deeper culturally grounded reasoning from generative RMs via reinforcement learning from verifiable rewards (RLVR) and employ well-designed rewards to ensure accurate preference judgments and high-quality structured evaluation criteria generation. Experimental results validate its efficacy in mitigating spurious features interference and advancing culture-aware reward modeling.