Test-Time Alignment of Text-to-Image Diffusion Models via Null-Text Embedding Optimisation
This addresses the challenge of aligning diffusion models to specific rewards during inference without reward hacking, which is crucial for applications requiring reliable and semantically coherent image generation.
The paper tackled the problem of test-time alignment in text-to-image diffusion models, where existing methods often under-optimise or over-optimise rewards, and proposed Null-TTA by optimising the unconditional embedding in classifier-free guidance, achieving state-of-the-art alignment while maintaining strong cross-reward generalisation.
Test-time alignment (TTA) aims to adapt models to specific rewards during inference. However, existing methods tend to either under-optimise or over-optimise (reward hack) the target reward function. We propose Null-Text Test-Time Alignment (Null-TTA), which aligns diffusion models by optimising the unconditional embedding in classifier-free guidance, rather than manipulating latent or noise variables. Due to the structured semantic nature of the text embedding space, this ensures alignment occurs on a semantically coherent manifold and prevents reward hacking (exploiting non-semantic noise patterns to improve the reward). Since the unconditional embedding in classifier-free guidance serves as the anchor for the model's generative distribution, Null-TTA directly steers model's generative distribution towards the target reward rather than just adjusting the samples, even without updating model parameters. Thanks to these desirable properties, we show that Null-TTA achieves state-of-the-art target test-time alignment while maintaining strong cross-reward generalisation. This establishes semantic-space optimisation as an effective and principled novel paradigm for TTA.