Transforming and Combining Rewards for Aligning Large Language Models
This work addresses a key challenge in aligning language models to multiple human preferences, offering a method to mitigate underfitting and reward hacking, which is incremental but impactful for improving RLHF techniques.
The paper tackles the problem of aligning large language models to human preferences by studying how to transform and combine reward models, proposing a log-sigmoid-centered transformation that emphasizes improving poorly-performing outputs and enables principled aggregation. Experiments show substantial improvements over baseline methods in aligning models to be both helpful and harmless.
A common approach for aligning language models to human preferences is to first learn a reward model from preference data, and then use this reward model to update the language model. We study two closely related problems that arise in this approach. First, any monotone transformation of the reward model preserves preference ranking; is there a choice that is ``better'' than others? Second, we often wish to align language models to multiple properties: how should we combine multiple reward models? Using a probabilistic interpretation of the alignment procedure, we identify a natural choice for transformation for (the common case of) rewards learned from Bradley-Terry preference models. The derived transformation is straightforward: we apply a log-sigmoid function to the centered rewards, a method we term ``LSC-transformation'' (log-sigmoid-centered transformation). This transformation has two important properties. First, it emphasizes improving poorly-performing outputs, rather than outputs that already score well. This mitigates both underfitting (where some prompts are not improved) and reward hacking (where the model learns to exploit misspecification of the reward model). Second, it enables principled aggregation of rewards by linking summation to logical conjunction: the sum of transformed rewards corresponds to the probability that the output is ``good'' in all measured properties, in a sense we make precise. Experiments aligning language models to be both helpful and harmless using RLHF show substantial improvements over the baseline (non-transformed) approach.