A Minimalist Method for Fine-tuning Text-to-Image Diffusion Models
This work addresses the complexity in existing fine-tuning methods for diffusion models, offering a simpler approach for researchers and practitioners in generative AI.
The paper tackles the problem of fine-tuning text-to-image diffusion models by introducing Noise PPO, a minimalist reinforcement learning algorithm that learns a prompt-conditioned initial noise generator, which improves text-image alignment and sample quality, with the most significant gains observed at low inference steps.
Recent work uses reinforcement learning (RL) to fine-tune text-to-image diffusion models, improving text-image alignment and sample quality. However, existing approaches introduce unnecessary complexity: they cache the full sampling trajectory, depend on differentiable reward models or large preference datasets, or require specialized guidance techniques. Motivated by the "golden noise" hypothesis -- that certain initial noise samples can consistently yield superior alignment -- we introduce Noise PPO, a minimalist RL algorithm that leaves the pre-trained diffusion model entirely frozen and learns a prompt-conditioned initial noise generator. Our approach requires no trajectory storage, reward backpropagation, or complex guidance tricks. Extensive experiments show that optimizing the initial noise distribution consistently improves alignment and sample quality over the original model, with the most significant gains at low inference steps. As the number of inference steps increases, the benefit of noise optimization diminishes but remains present. These findings clarify the scope and limitations of the golden noise hypothesis and reinforce the practical value of minimalist RL fine-tuning for diffusion models.