Mitigating Length Bias in RLHF through a Causal Lens
This work addresses a specific bottleneck in RLHF for aligning LLMs with human preferences, offering an incremental improvement to enhance reward modeling robustness.
The paper tackled the problem of length bias in RLHF-trained reward models, which favor longer responses by conflating verbosity with quality, and proposed a causal framework with counterfactual data augmentation to mitigate this bias, resulting in reduced length bias and more concise, content-focused outputs from the policy model.
Reinforcement learning from human feedback (RLHF) is widely used to align large language models (LLMs) with human preferences. However, RLHF-trained reward models often exhibit length bias -- a systematic tendency to favor longer responses by conflating verbosity with quality. We propose a causal framework for analyzing and mitigating length bias in RLHF reward modeling. Central to our approach is a counterfactual data augmentation method that generates response pairs designed to isolate content quality from verbosity. These counterfactual examples are then used to train the reward model, enabling it to assess responses based on content quality independently of verbosity. Specifically, we construct (1) length-divergent pairs with similar content and (2) content-divergent pairs of similar length. Empirical evaluations show that our method reduces length bias in reward assignment and leads to more concise, content-focused outputs from the policy model. These findings demonstrate that the proposed approach effectively reduces length bias and improves the robustness and content sensitivity of reward modeling in RLHF pipelines.