Mitigating Preference Hacking in Policy Optimization with Pessimism
This addresses the problem of preference hacking in RLHF for AI alignment, offering a robust solution to improve model reliability, though it appears incremental as it builds on existing RLHF methods with a novel pessimistic twist.
The paper tackles overoptimization in reinforcement learning from human feedback (RLHF) by proposing pessimistic objectives that are provably robust, leading to algorithms like P3O and PRPO that show remarkable resilience in tasks such as fine-tuning language models for document summarization and creating helpful assistants.
This work tackles the problem of overoptimization in reinforcement learning from human feedback (RLHF), a prevalent technique for aligning models with human preferences. RLHF relies on reward or preference models trained on \emph{fixed preference datasets}, and these models are unreliable when evaluated outside the support of this preference data, leading to the common reward or preference hacking phenomenon. We propose novel, pessimistic objectives for RLHF which are provably robust to overoptimization through the use of pessimism in the face of uncertainty, and design practical algorithms, P3O and PRPO, to optimize these objectives. Our approach is derived for the general preference optimization setting, but can be used with reward models as well. We evaluate P3O and PRPO on the tasks of fine-tuning language models for document summarization and creating helpful assistants, demonstrating remarkable resilience to overoptimization.