NoiseTilt: Noise-Tilted Reverse Kernels for Diffusion Reward Alignment
This work addresses the trade-off between gradient-based guidance and search-based methods in reward-guided diffusion sampling, offering a solution that maintains both quality and gradient signal for practitioners using pretrained diffusion models.
NoiseTilt introduces the Noise-Tilted Reverse Kernel (NTRK) for reward-guided diffusion sampling, which injects reward gradients through the noise term while keeping the pretrained reverse kernel unchanged. NTRK achieves state-of-the-art reward alignment without sacrificing sample quality, and on aesthetic generation it matches the reward of the best baseline at 500 NFEs using only 25 NFEs, a 20x reduction in compute.
We introduce the Noise-Tilted Reverse Kernel (NTRK), a reward-guided diffusion sampler that injects reward gradients through the noise term, leaving the pretrained reverse kernel unchanged and requiring only a single sample per step. Reward-guided sampling at inference time has greatly expanded the versatility of pretrained diffusion models. Yet existing methods face a trade-off. Gradient-based guidance shifts the reverse mean, steering generation but pushing intermediate states outside the region that the model was trained on and degrading quality. Search-based methods preserve quality but gain no gradient signal. No prior method achieves both. NTRK resolves this by keeping the reverse mean fixed and biasing the noise term toward high reward. We introduce a whitening operator, the central mechanism behind NTRK, that makes the reward gradient safe to inject as noise without losing its guiding signal. Across various reward alignment tasks, NTRK outperforms recent state-of-the-art baselines without losing sample quality. Remarkably, on aesthetic generation, NTRK surpasses the reward of the best baseline at 500 NFEs using only 25 NFEs, a 20$\times$ reduction in compute.