Margin-aware Preference Optimization for Aligning Diffusion Models without Reference
This addresses a fundamental limitation in preference alignment for text-to-image diffusion models, offering a more efficient solution for tasks like style adaptation and personalization, though it is incremental as it builds on existing preference optimization frameworks.
The paper tackles the problem of 'reference mismatch' in aligning text-to-image diffusion models, which hinders adaptation to new styles or objects, and introduces MaPO, a reference-agnostic method that outperforms existing approaches like DPO and DreamBooth while reducing training time by 15%.
Modern preference alignment methods, such as DPO, rely on divergence regularization to a reference model for training stability-but this creates a fundamental problem we call "reference mismatch." In this paper, we investigate the negative impacts of reference mismatch in aligning text-to-image (T2I) diffusion models, showing that larger reference mismatch hinders effective adaptation given the same amount of data, e.g., as when learning new artistic styles, or personalizing to specific objects. We demonstrate this phenomenon across text-to-image (T2I) diffusion models and introduce margin-aware preference optimization (MaPO), a reference-agnostic approach that breaks free from this constraint. By directly optimizing the likelihood margin between preferred and dispreferred outputs under the Bradley-Terry model without anchoring to a reference, MaPO transforms diverse T2I tasks into unified pairwise preference optimization. We validate MaPO's versatility across five challenging domains: (1) safe generation, (2) style adaptation, (3) cultural representation, (4) personalization, and (5) general preference alignment. Our results reveal that MaPO's advantage grows dramatically with reference mismatch severity, outperforming both DPO and specialized methods like DreamBooth while reducing training time by 15%. MaPO thus emerges as a versatile and memory-efficient method for generic T2I adaptation tasks.