Trust-Region Noise Search for Black-Box Alignment of Diffusion and Flow Models
This addresses the need for efficient and versatile alignment of generative models for applications in creative and scientific domains, though it appears incremental as an optimization technique.
The paper tackles the problem of aligning diffusion and flow models to target rewards at inference time by proposing a trust-region based search algorithm (TRS) that optimizes source noise as a black-box approach, achieving significantly improved output samples across text-to-image, molecule, and protein design tasks compared to base models and other alignment methods.
Optimizing the noise samples of diffusion and flow models is an increasingly popular approach to align these models to target rewards at inference time. However, we observe that these approaches are usually restricted to differentiable or cheap reward models, the formulation of the underlying pretrained generative model, or are memory/compute inefficient. We instead propose a simple trust-region based search algorithm (TRS) which treats the pre-trained generative and reward models as a black-box and only optimizes the source noise. Our approach achieves a good balance between global exploration and local exploitation, and is versatile and easily adaptable to various generative settings and reward models with minimal hyperparameter tuning. We evaluate TRS across text-to-image, molecule and protein design tasks, and obtain significantly improved output samples over the base generative models and other inference-time alignment approaches which optimize the source noise sample, or even the entire reverse-time sampling noise trajectories in the case of diffusion models. Our source code is publicly available.