Towards Reward Fairness in RLHF: From a Resource Allocation Perspective
This addresses fairness issues in aligning large language models with human preferences, which is an incremental improvement in RLHF methodology.
The paper tackles the problem of reward unfairness in Reinforcement Learning from Human Feedback (RLHF) by proposing a bias-agnostic method that models preference learning as a resource allocation problem, and experiments show it aligns large language models with human preferences more fairly.
Rewards serve as proxies for human preferences and play a crucial role in Reinforcement Learning from Human Feedback (RLHF). However, if these rewards are inherently imperfect, exhibiting various biases, they can adversely affect the alignment of large language models (LLMs). In this paper, we collectively define the various biases present in rewards as the problem of reward unfairness. We propose a bias-agnostic method to address the issue of reward fairness from a resource allocation perspective, without specifically designing for each type of bias, yet effectively mitigating them. Specifically, we model preference learning as a resource allocation problem, treating rewards as resources to be allocated while considering the trade-off between utility and fairness in their distribution. We propose two methods, Fairness Regularization and Fairness Coefficient, to achieve fairness in rewards. We apply our methods in both verification and reinforcement learning scenarios to obtain a fairness reward model and a policy model, respectively. Experiments conducted in these scenarios demonstrate that our approach aligns LLMs with human preferences in a more fair manner.