DreamBoothDPO: Improving Personalized Generation using Direct Preference Optimization
This work addresses a challenging open problem in personalized image generation for users seeking better control over concept injection, though it appears incremental as it builds on existing DPO and fine-tuning techniques.
The paper tackles the problem of balancing concept fidelity with contextual alignment in personalized diffusion models for text-to-image generation by proposing an RL-based approach that uses synthetic paired datasets for DPO-like training, eliminating the need for human annotations and outperforming a naive baseline in convergence speed and output quality.
Personalized diffusion models have shown remarkable success in Text-to-Image (T2I) generation by enabling the injection of user-defined concepts into diverse contexts. However, balancing concept fidelity with contextual alignment remains a challenging open problem. In this work, we propose an RL-based approach that leverages the diverse outputs of T2I models to address this issue. Our method eliminates the need for human-annotated scores by generating a synthetic paired dataset for DPO-like training using external quality metrics. These better-worse pairs are specifically constructed to improve both concept fidelity and prompt adherence. Moreover, our approach supports flexible adjustment of the trade-off between image fidelity and textual alignment. Through multi-step training, our approach outperforms a naive baseline in convergence speed and output quality. We conduct extensive qualitative and quantitative analysis, demonstrating the effectiveness of our method across various architectures and fine-tuning techniques. The source code can be found at https://github.com/ControlGenAI/DreamBoothDPO.