Direct Diffusion Score Preference Optimization via Stepwise Contrastive Policy-Pair Supervision
This work addresses the challenge of training diffusion models for better alignment and quality without costly human-labeled data, offering a practical solution for generative AI applications.
The authors tackled the problem of aligning diffusion model outputs with user intent and aesthetic quality by introducing Direct Diffusion Score Preference Optimization (DDSPO), which uses stepwise contrastive supervision from policy pairs to improve text-image alignment and visual quality, outperforming or matching existing methods with less supervision.
Diffusion models have achieved impressive results in generative tasks such as text-to-image synthesis, yet they often struggle to fully align outputs with nuanced user intent and maintain consistent aesthetic quality. Existing preference-based training methods like Diffusion Direct Preference Optimization help address these issues but rely on costly and potentially noisy human-labeled datasets. In this work, we introduce Direct Diffusion Score Preference Optimization (DDSPO), which directly derives per-timestep supervision from winning and losing policies when such policies are available. Unlike prior methods that operate solely on final samples, DDSPO provides dense, transition-level signals across the denoising trajectory. In practice, we avoid reliance on labeled data by automatically generating preference signals using a pretrained reference model: we contrast its outputs when conditioned on original prompts versus semantically degraded variants. This practical strategy enables effective score-space preference supervision without explicit reward modeling or manual annotations. Empirical results demonstrate that DDSPO improves text-image alignment and visual quality, outperforming or matching existing preference-based methods while requiring significantly less supervision. Our implementation is available at: https://dohyun-as.github.io/DDSPO