End-to-End Diffusion Latent Optimization Improves Classifier Guidance
This work addresses the challenge of precise creative control in image generation and editing for users of diffusion models, representing an incremental improvement over existing guidance methods.
The paper tackled the problem of suboptimal control in classifier guidance for diffusion models by proposing DOODL, a method that optimizes diffusion latents using gradients from a pre-trained classifier on true generated pixels, which outperformed one-step classifier guidance on computational and human evaluation metrics across various tasks.
Classifier guidance -- using the gradients of an image classifier to steer the generations of a diffusion model -- has the potential to dramatically expand the creative control over image generation and editing. However, currently classifier guidance requires either training new noise-aware models to obtain accurate gradients or using a one-step denoising approximation of the final generation, which leads to misaligned gradients and sub-optimal control. We highlight this approximation's shortcomings and propose a novel guidance method: Direct Optimization of Diffusion Latents (DOODL), which enables plug-and-play guidance by optimizing diffusion latents w.r.t. the gradients of a pre-trained classifier on the true generated pixels, using an invertible diffusion process to achieve memory-efficient backpropagation. Showcasing the potential of more precise guidance, DOODL outperforms one-step classifier guidance on computational and human evaluation metrics across different forms of guidance: using CLIP guidance to improve generations of complex prompts from DrawBench, using fine-grained visual classifiers to expand the vocabulary of Stable Diffusion, enabling image-conditioned generation with a CLIP visual encoder, and improving image aesthetics using an aesthetic scoring network. Code at https://github.com/salesforce/DOODL.