SEE-DPO: Self Entropy Enhanced Direct Preference Optimization
This work addresses stability and quality issues in aligning diffusion models with human preferences, representing an incremental improvement over existing DPO methods.
The paper tackled overfitting and reward hacking in DPO-based methods for text-to-image diffusion models by introducing a self-entropy regularization mechanism, achieving state-of-the-art results on key image generation metrics.
Direct Preference Optimization (DPO) has been successfully used to align large language models (LLMs) according to human preferences, and more recently it has also been applied to improving the quality of text-to-image diffusion models. However, DPO-based methods such as SPO, Diffusion-DPO, and D3PO are highly susceptible to overfitting and reward hacking, especially when the generative model is optimized to fit out-of-distribution during prolonged training. To overcome these challenges and stabilize the training of diffusion models, we introduce a self-entropy regularization mechanism in reinforcement learning from human feedback. This enhancement improves DPO training by encouraging broader exploration and greater robustness. Our regularization technique effectively mitigates reward hacking, leading to improved stability and enhanced image quality across the latent space. Extensive experiments demonstrate that integrating human feedback with self-entropy regularization can significantly boost image diversity and specificity, achieving state-of-the-art results on key image generation metrics.