Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biases
This work addresses a critical problem for practitioners using diffusion models in generative workflows, though it appears incremental as it builds on existing alignment strategies.
The paper tackles reward overoptimization in aligning diffusion models with human preferences by identifying mismatches with temporal inductive biases and primacy biases from active neurons, and proposes TDPO-R, which demonstrates superior efficacy in mitigating this issue.
Bridging the gap between diffusion models and human preferences is crucial for their integration into practical generative workflows. While optimizing downstream reward models has emerged as a promising alignment strategy, concerns arise regarding the risk of excessive optimization with learned reward models, which potentially compromises ground-truth performance. In this work, we confront the reward overoptimization problem in diffusion model alignment through the lenses of both inductive and primacy biases. We first identify a mismatch between current methods and the temporal inductive bias inherent in the multi-step denoising process of diffusion models, as a potential source of reward overoptimization. Then, we surprisingly discover that dormant neurons in our critic model act as a regularization against reward overoptimization while active neurons reflect primacy bias. Motivated by these observations, we propose Temporal Diffusion Policy Optimization with critic active neuron Reset (TDPO-R), a policy gradient algorithm that exploits the temporal inductive bias of diffusion models and mitigates the primacy bias stemming from active neurons. Empirical results demonstrate the superior efficacy of our methods in mitigating reward overoptimization. Code is avaliable at https://github.com/ZiyiZhang27/tdpo.