Iterative Tilting for Diffusion Fine-Tuning
This work addresses a specific technical challenge in fine-tuning diffusion models for researchers in machine learning, but it appears incremental as it builds on existing methods without demonstrating broad applicability or significant real-world impact.
The authors tackled the problem of fine-tuning diffusion models toward reward-tilted distributions by introducing iterative tilting, a gradient-free method that decomposes large tilts into smaller sequential updates, requiring only forward reward evaluations and avoiding backpropagation through sampling chains. They validated the method on a two-dimensional Gaussian mixture with linear reward, where the exact tilted distribution is available in closed form, but no concrete numerical results or performance metrics were provided.
We introduce iterative tilting, a gradient-free method for fine-tuning diffusion models toward reward-tilted distributions. The method decomposes a large reward tilt $\exp(λr)$ into $N$ sequential smaller tilts, each admitting a tractable score update via first-order Taylor expansion. This requires only forward evaluations of the reward function and avoids backpropagating through sampling chains. We validate on a two-dimensional Gaussian mixture with linear reward, where the exact tilted distribution is available in closed form.