InstructEngine: Instruction-driven Text-to-Image Alignment
This work addresses scaling and efficiency issues in text-to-image alignment for AI researchers and practitioners, though it appears incremental by building on existing RLHF/RLAIF approaches.
The paper tackles the problem of high annotation costs and algorithmic inefficiencies in text-to-image alignment methods by proposing InstructEngine, which uses an automated data construction pipeline to generate 25K preference pairs and a cross-validation alignment method, resulting in performance improvements of 10.53% for SD v1.5 and 5.30% for SDXL on DrawBench.
Reinforcement Learning from Human/AI Feedback (RLHF/RLAIF) has been extensively utilized for preference alignment of text-to-image models. Existing methods face certain limitations in terms of both data and algorithm. For training data, most approaches rely on manual annotated preference data, either by directly fine-tuning the generators or by training reward models to provide training signals. However, the high annotation cost makes them difficult to scale up, the reward model consumes extra computation and cannot guarantee accuracy. From an algorithmic perspective, most methods neglect the value of text and only take the image feedback as a comparative signal, which is inefficient and sparse. To alleviate these drawbacks, we propose the InstructEngine framework. Regarding annotation cost, we first construct a taxonomy for text-to-image generation, then develop an automated data construction pipeline based on it. Leveraging advanced large multimodal models and human-defined rules, we generate 25K text-image preference pairs. Finally, we introduce cross-validation alignment method, which refines data efficiency by organizing semantically analogous samples into mutually comparable pairs. Evaluations on DrawBench demonstrate that InstructEngine improves SD v1.5 and SDXL's performance by 10.53% and 5.30%, outperforming state-of-the-art baselines, with ablation study confirming the benefits of InstructEngine's all components. A win rate of over 50% in human reviews also proves that InstructEngine better aligns with human preferences.