Explaining and Preventing Alignment Collapse in Iterative RLHF
Addresses a fundamental flaw in iterative RLHF for AI alignment, preventing reward hacking in LLM training.
Iterative RLHF suffers from alignment collapse where the policy exploits reward model blind spots, producing low-quality outputs. The proposed FPO method prevents this collapse by regularizing the policy's influence on RM updates, demonstrated on Llama-3.2-1B.
Reinforcement learning from human feedback (RLHF) typically assumes a static or non-strategic reward model (RM). In iterative deployment, however, the policy generates the data on which the RM is retrained, creating a feedback loop. Building on the Stackelberg game formulation of this interaction, we derive an analytical decomposition of the policy's true optimization gradient into a standard policy gradient and a parameter-steering term that captures the policy's influence on the RM's future parameters. We show that standard iterative RLHF, which drops this steering term entirely, suffers from alignment collapse: the policy systematically exploits the RM's blind spots, producing low-quality, high-reward outputs whose feedback reinforces the very errors it exploits. To mitigate this, we propose foresighted policy optimization (FPO), a mechanism-design intervention that restores the missing steering term by regularizing the policy's parameter-steering effect on RM updates. We instantiate FPO via a scalable first-order approximation and demonstrate that it prevents alignment collapse on both controlled environments and an LLM alignment pipeline using Llama-3.2-1B.