Multi-Reward GRPO Fine-Tuning for De-biasing Large Language Models: A Study Based on Chinese-Context Discrimination Data
This work addresses the issue of culturally specific and multi-dimensional discrimination in LLMs for users and developers, offering a replicable framework for ethical alignment, though it is incremental as it builds on existing alignment techniques like RLHF and DPO.
The paper tackled the problem of implicit biases and discriminatory tendencies in Large Language Models (LLMs), particularly in culturally specific contexts like Chinese discrimination categories, by proposing a Multi-Reward Group Relative Policy Optimization (GRPO) framework, which resulted in significant reductions in bias intensity and improved alignment with non-discriminatory standards without compromising fluency or informativeness.
Large Language Models (LLMs) often exhibit implicit biases and discriminatory tendencies that reflect underlying social stereotypes. While recent alignment techniques such as RLHF and DPO have mitigated some of these issues, they remain limited in addressing culturally specific and multi-dimensional forms of discrimination. This paper proposes a Multi-Reward Group Relative Policy Optimization (GRPO) framework to fine-tune LLMs toward ethical and bias-free behavior. Our approach constructs a synthetic English-language dataset derived from Chinese-context discrimination categories, including regional, ethnic, and occupational biases. Each instance is paired with both neutral and biased responses to train a reward model based on DeBERTa-v3, which provides multi-dimensional reward signals capturing fairness, neutrality, and linguistic quality. The trained reward model then guides GRPO fine-tuning to optimize model outputs along these ethical dimensions. Experimental results demonstrate significant reductions in bias intensity and improved alignment with non-discriminatory standards without compromising fluency or informativeness. This study highlights the effectiveness of GRPO-based multi-reward optimization for de-biasing LLMs and offers a replicable framework for cultural-contextual ethical alignment.