Elucidating Optimal Reward-Diversity Tradeoffs in Text-to-Image Diffusion Models
This addresses a key challenge in aligning text-to-image models with human preferences without sacrificing diversity, which is crucial for practical applications in creative and safe AI-generated content.
The paper tackles the problem of reward hacking in text-to-image diffusion models, where fine-tuning for human preferences reduces image diversity, and introduces Annealed Importance Guidance (AIG) to achieve Pareto-optimal tradeoffs between reward optimization and diversity, with experiments and a user study confirming improved diversity and quality.
Text-to-image (T2I) diffusion models have become prominent tools for generating high-fidelity images from text prompts. However, when trained on unfiltered internet data, these models can produce unsafe, incorrect, or stylistically undesirable images that are not aligned with human preferences. To address this, recent approaches have incorporated human preference datasets to fine-tune T2I models or to optimize reward functions that capture these preferences. Although effective, these methods are vulnerable to reward hacking, where the model overfits to the reward function, leading to a loss of diversity in the generated images. In this paper, we prove the inevitability of reward hacking and study natural regularization techniques like KL divergence and LoRA scaling, and their limitations for diffusion models. We also introduce Annealed Importance Guidance (AIG), an inference-time regularization inspired by Annealed Importance Sampling, which retains the diversity of the base model while achieving Pareto-Optimal reward-diversity tradeoffs. Our experiments demonstrate the benefits of AIG for Stable Diffusion models, striking the optimal balance between reward optimization and image diversity. Furthermore, a user study confirms that AIG improves diversity and quality of generated images across different model architectures and reward functions.