Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion Models
This work addresses alignment challenges in text-to-image diffusion models, offering a model-agnostic solution that is incremental but provides consistent improvements for AI-generated content applications.
The paper tackled the problem of aligning text-to-image diffusion models with human preferences by identifying a pathology in existing methods where increasing preference margins degrades generation quality, and introduced Diffusion-SDPO, a safeguarded update rule that improved performance on benchmarks with gains over baselines in automated preference, aesthetic, and prompt alignment metrics.
Text-to-image diffusion models deliver high-quality images, yet aligning them with human preferences remains challenging. We revisit diffusion-based Direct Preference Optimization (DPO) for these models and identify a critical pathology: enlarging the preference margin does not necessarily improve generation quality. In particular, the standard Diffusion-DPO objective can increase the reconstruction error of both winner and loser branches. Consequently, degradation of the less-preferred outputs can become sufficiently severe that the preferred branch is also adversely affected even as the margin grows. To address this, we introduce Diffusion-SDPO, a safeguarded update rule that preserves the winner by adaptively scaling the loser gradient according to its alignment with the winner gradient. A first-order analysis yields a closed-form scaling coefficient that guarantees the error of the preferred output is non-increasing at each optimization step. Our method is simple, model-agnostic, broadly compatible with existing DPO-style alignment frameworks and adds only marginal computational overhead. Across standard text-to-image benchmarks, Diffusion-SDPO delivers consistent gains over preference-learning baselines on automated preference, aesthetic, and prompt alignment metrics. Code is publicly available at https://github.com/AIDC-AI/Diffusion-SDPO.