Rethinking Direct Preference Optimization in Diffusion Models
This work addresses a critical challenge in text-to-image generation for users needing more human-aligned outputs, though it is incremental as it builds on existing preference optimization methods.
The paper tackles the problem of aligning text-to-image diffusion models with human preferences by addressing limited exploration and reward scale imbalance, resulting in improved performance on human preference evaluation benchmarks.
Aligning text-to-image (T2I) diffusion models with human preferences has emerged as a critical research challenge. While recent advances in this area have extended preference optimization techniques from large language models (LLMs) to the diffusion setting, they often struggle with limited exploration. In this work, we propose a novel and orthogonal approach to enhancing diffusion-based preference optimization. First, we introduce a stable reference model update strategy that relaxes the frozen reference model, encouraging exploration while maintaining a stable optimization anchor through reference model regularization. Second, we present a timestep-aware training strategy that mitigates the reward scale imbalance problem across timesteps. Our method can be integrated into various preference optimization algorithms. Experimental results show that our approach improves the performance of state-of-the-art methods on human preference evaluation benchmarks.