Inference-Time Attribute Distribution Alignment for Unconditional Diffusion
For practitioners using pretrained diffusion models, this enables population-level attribute control (e.g., fairness) at inference time, but the method is incremental as it adapts existing optimal control techniques to a new problem setting.
The paper tackles inference-time control of attribute distributions in unconditional diffusion models, proposing an optimal control approach that aligns sample populations to target distributions (e.g., demographic balance) without retraining. Experiments show improved alignment over baselines while preserving data fidelity.
Inference-time controllable generation is essential for real-world applications of unconditional diffusion models. However, most existing techniques focus on individual samples, struggling in applications that require the sample population to follow specific attribute distributions (e.g., demographic balance or semantic proportions). We formalize this setting as the inference-time attribute distributional alignment problem for pretrained unconditional diffusion models. To address this, we cast inference-time attribute distributional alignment as an optimal control problem over the reverse diffusion process, viewing the process as the rollout of a dynamical system and augmenting it with additive, time-dependent perturbations as control. We solve for the perturbations using an optimal-control-based algorithm to optimize a differentiable distribution-matching objective while penalizing control effort to preserve data fidelity. Experiment results in image generation demonstrate that our proposed plug-and-play approach can better align attribute distributions to diverse and flexible test-time targets compared to baselines, without retraining or finetuning the pretrained diffusion model.