Temporal Score Rescaling for Temperature Sampling in Diffusion and Flow Models
This work addresses the need for flexible sampling control in generative models across various domains, but it is incremental as it builds on existing score-based methods without altering training.
The authors tackled the problem of controlling sampling diversity in diffusion and flow models by introducing a temporal score rescaling mechanism that allows users to sample from sharper or broader distributions without retraining. They demonstrated performance gains across five tasks, such as improved depth prediction with sharper sampling and better image generation with flatter sampling.
We present a mechanism to steer the sampling diversity of denoising diffusion and flow matching models, allowing users to sample from a sharper or broader distribution than the training distribution. We build on the observation that these models leverage (learned) score functions of noisy data distributions for sampling and show that rescaling these allows one to effectively control a `local' sampling temperature. Notably, this approach does not require any finetuning or alterations to training strategy, and can be applied to any off-the-shelf model and is compatible with both deterministic and stochastic samplers. We first validate our framework on toy 2D data, and then demonstrate its application for diffusion models trained across five disparate tasks -- image generation, pose estimation, depth prediction, robot manipulation, and protein design. We find that across these tasks, our approach allows sampling from sharper (or flatter) distributions, yielding performance gains e.g., depth prediction models benefit from sampling more likely depth estimates, whereas image generation models perform better when sampling a slightly flatter distribution. Project page: https://temporalscorerescaling.github.io