AGFSync: Leveraging AI-Generated Feedback for Preference Optimization in Text-to-Image Generation
This work addresses the problem of costly labeled data acquisition for refining text-to-image models, offering a scalable alignment technique for AI researchers and practitioners, though it is incremental as it builds on existing methods like DPO and VLMs.
The paper tackles challenges in text-to-image diffusion models, such as prompt-following and image quality, by introducing AGFSync, a framework that uses AI-generated feedback for preference optimization, resulting in notable improvements in VQA scores, aesthetic evaluations, and HPSv2 benchmark performance over base models like SD v1.4, v1.5, and SDXL-base.
Text-to-Image (T2I) diffusion models have achieved remarkable success in image generation. Despite their progress, challenges remain in both prompt-following ability, image quality and lack of high-quality datasets, which are essential for refining these models. As acquiring labeled data is costly, we introduce AGFSync, a framework that enhances T2I diffusion models through Direct Preference Optimization (DPO) in a fully AI-driven approach. AGFSync utilizes Vision-Language Models (VLM) to assess image quality across style, coherence, and aesthetics, generating feedback data within an AI-driven loop. By applying AGFSync to leading T2I models such as SD v1.4, v1.5, and SDXL-base, our extensive experiments on the TIFA dataset demonstrate notable improvements in VQA scores, aesthetic evaluations, and performance on the HPSv2 benchmark, consistently outperforming the base models. AGFSync's method of refining T2I diffusion models paves the way for scalable alignment techniques. Our code and dataset are publicly available at https://anjingkun.github.io/AGFSync.