SAIL: Self-Amplified Iterative Learning for Diffusion Model Alignment with Minimal Human Feedback
This addresses the problem of costly human feedback for AI researchers and practitioners in diffusion model alignment, offering a novel method that reduces data needs, though it is incremental in leveraging self-improvement within existing models.
The paper tackles the challenge of aligning diffusion models with human preferences when reward models are unavailable and large-scale preference data is expensive, proposing SAIL, a self-amplified iterative learning framework that uses minimal human feedback. The result shows that SAIL outperforms state-of-the-art methods across multiple benchmarks while using only 6% of the preference data required by existing approaches.
Aligning diffusion models with human preferences remains challenging, particularly when reward models are unavailable or impractical to obtain, and collecting large-scale preference datasets is prohibitively expensive. \textit{This raises a fundamental question: can we achieve effective alignment using only minimal human feedback, without auxiliary reward models, by unlocking the latent capabilities within diffusion models themselves?} In this paper, we propose \textbf{SAIL} (\textbf{S}elf-\textbf{A}mplified \textbf{I}terative \textbf{L}earning), a novel framework that enables diffusion models to act as their own teachers through iterative self-improvement. Starting from a minimal seed set of human-annotated preference pairs, SAIL operates in a closed-loop manner where the model progressively generates diverse samples, self-annotates preferences based on its evolving understanding, and refines itself using this self-augmented dataset. To ensure robust learning and prevent catastrophic forgetting, we introduce a ranked preference mixup strategy that carefully balances exploration with adherence to initial human priors. Extensive experiments demonstrate that SAIL consistently outperforms state-of-the-art methods across multiple benchmarks while using merely 6\% of the preference data required by existing approaches, revealing that diffusion models possess remarkable self-improvement capabilities that, when properly harnessed, can effectively replace both large-scale human annotation and external reward models.