Spectral Guidance for Flexible and Efficient Control of Diffusion Models
Provides a flexible and efficient framework for controlling diffusion models without retraining or backpropagation, addressing the need for practical guidance in generative AI.
Spectral Guidance enables efficient, training-free control of diffusion models by projecting guidance signals onto a learned basis of singular functions, achieving 37 percentage point improvement in conditional accuracy on CIFAR-10 over training-free baselines with 4x faster sampling.
We introduce Spectral Guidance, a framework for controlling diffusion models by leveraging the intrinsic geometry of the generative process. As data is progressively corrupted by noise, only a small number of features remain informative for control. We characterize them as the singular functions of a conditional expectation operator and show that they can be learned via a self-supervised objective. Once recovered, this basis enables the projection of arbitrary guidance signals, such as labels, CLIP embeddings, or masks, directly onto the sampling trajectory. This approach allows for stable, high-fidelity control without retraining or denoiser backpropagation during sampling. Empirically, we improve conditional accuracy on CIFAR-10 by 37 percentage points over the strongest training-free baseline while offering $4\times$ faster sampling. Moreover, the same representations that support label and CLIP guidance also enable spatial control, such as mask-based guidance, without auxiliary models. Finally, our framework reveals a phase transition in the generative process, pinpointing the optimal time window for effective guidance.