Data-regularized Reinforcement Learning for Diffusion Models at Scale
This work addresses the challenge of reward hacking in RL for diffusion models, which is critical for improving alignment with human preferences in generative AI, though it appears incremental as it builds on existing regularization methods.
The paper tackles the problem of aligning generative diffusion models with human preferences via reinforcement learning, which often suffers from reward hacking issues like quality degradation and reduced diversity. The proposed Data-regularized Diffusion Reinforcement Learning (DDRL) framework significantly improves rewards and alleviates reward hacking, as demonstrated through over a million GPU hours of experiments and ten thousand double-blind human evaluations on high-resolution video generation tasks.
Aligning generative diffusion models with human preferences via reinforcement learning (RL) is critical yet challenging. Most existing algorithms are often vulnerable to reward hacking, such as quality degradation, over-stylization, or reduced diversity. Our analysis demonstrates that this can be attributed to the inherent limitations of their regularization, which provides unreliable penalties. We introduce Data-regularized Diffusion Reinforcement Learning (DDRL), a novel framework that uses the forward KL divergence to anchor the policy to an off-policy data distribution. Theoretically, DDRL enables robust, unbiased integration of RL with standard diffusion training. Empirically, this translates into a simple yet effective algorithm that combines reward maximization with diffusion loss minimization. With over a million GPU hours of experiments and ten thousand double-blind human evaluations, we demonstrate on high-resolution video generation tasks that DDRL significantly improves rewards while alleviating the reward hacking seen in baselines, achieving the highest human preference and establishing a robust and scalable paradigm for diffusion post-training.