Readout Guidance: Learning Control from Diffusion Features
This provides a convenient and unified framework for conditional image generation, which is incremental but simplifies control tasks like drag-based manipulation and identity consistency.
The authors tackled the problem of controlling text-to-image diffusion models by introducing Readout Guidance, a method that uses lightweight readout heads to extract signals from frozen diffusion features for conditional generation, resulting in significantly fewer parameters and training samples needed compared to prior methods.
We present Readout Guidance, a method for controlling text-to-image diffusion models with learned signals. Readout Guidance uses readout heads, lightweight networks trained to extract signals from the features of a pre-trained, frozen diffusion model at every timestep. These readouts can encode single-image properties, such as pose, depth, and edges; or higher-order properties that relate multiple images, such as correspondence and appearance similarity. Furthermore, by comparing the readout estimates to a user-defined target, and back-propagating the gradient through the readout head, these estimates can be used to guide the sampling process. Compared to prior methods for conditional generation, Readout Guidance requires significantly fewer added parameters and training samples, and offers a convenient and simple recipe for reproducing different forms of conditional control under a single framework, with a single architecture and sampling procedure. We showcase these benefits in the applications of drag-based manipulation, identity-consistent generation, and spatially aligned control. Project page: https://readout-guidance.github.io.