DeDPO: Debiased Direct Preference Optimization for Diffusion Models
This provides a scalable solution for human-AI alignment in diffusion models, addressing a key bottleneck in off-policy training with cost-effective synthetic supervision.
The paper tackles the high cost and scalability bottleneck of Direct Preference Optimization (DPO) for diffusion models by proposing DeDPO, a semi-supervised framework that augments limited human data with synthetic AI feedback and integrates debiased estimation to correct bias and noise, achieving performance that matches or exceeds models trained on fully human-labeled data.
Direct Preference Optimization (DPO) has emerged as a predominant alignment method for diffusion models, facilitating off-policy training without explicit reward modeling. However, its reliance on large-scale, high-quality human preference labels presents a severe cost and scalability bottleneck. To overcome this, We propose a semi-supervised framework augmenting limited human data with a large corpus of unlabeled pairs annotated via cost-effective synthetic AI feedback. Our paper introduces Debiased DPO (DeDPO), which uniquely integrates a debiased estimation technique from causal inference into the DPO objective. By explicitly identifying and correcting the systematic bias and noise inherent in synthetic annotators, DeDPO ensures robust learning from imperfect feedback sources, including self-training and Vision-Language Models (VLMs). Experiments demonstrate that DeDPO is robust to the variations in synthetic labeling methods, achieving performance that matches and occasionally exceeds the theoretical upper bound of models trained on fully human-labeled data. This establishes DeDPO as a scalable solution for human-AI alignment using inexpensive synthetic supervision.