Bias Fitting to Mitigate Length Bias of Reward Model in RLHF
This addresses a specific reward hacking issue in aligning large language models with human preferences, offering an incremental improvement over previous methods that assumed linear bias or lacked characterization.
The paper tackles the problem of length bias in reward models used for Reinforcement Learning from Human Feedback (RLHF), where models favor longer responses regardless of quality, and proposes FiMi-RM, a framework that learns and corrects non-linear bias patterns, resulting in improved length-controlled win rates and reduced verbosity without performance loss.
Reinforcement Learning from Human Feedback relies on reward models to align large language models with human preferences. However, RLHF often suffers from reward hacking, wherein policy learning exploits flaws in the trained reward model to maximize reward scores without genuinely aligning with human preferences. A significant example of such reward hacking is length bias, where reward models usually favor longer responses irrespective of actual response quality. Previous works on length bias have notable limitations, these approaches either mitigate bias without characterizing the bias form, or simply assume a linear length-reward relation. To accurately model the intricate nature of length bias and facilitate more effective bias mitigation, we propose FiMi-RM (Bias Fitting to Mitigate Length Bias of Reward Model in RLHF), a framework that autonomously learns and corrects underlying bias patterns. Our approach consists of three stages: First, we train a standard reward model which inherently contains length bias. Next, we deploy a lightweight fitting model to explicitly capture the non-linear relation between length and reward. Finally, we incorporate this learned relation into the reward model to debias. Experimental results demonstrate that FiMi-RM achieves a more balanced length-reward distribution. Furthermore, when applied to alignment algorithms, our debiased reward model improves length-controlled win rate and reduces verbosity without compromising its performance.