Dynamic Negative Guidance of Diffusion Models
This work addresses the problem of improving control and safety in text-to-image generation for users of diffusion models, representing an incremental advancement over existing negative prompting techniques.
The paper tackled the limitation of constant guidance scale in negative prompting for diffusion models, which can cause suboptimal results, by introducing Dynamic Negative Guidance that modulates guidance based on time and state, leading to higher safety, preservation of class balance, and image quality in evaluations on MNIST, CIFAR10, and Stable Diffusion.
Negative Prompting (NP) is widely utilized in diffusion models, particularly in text-to-image applications, to prevent the generation of undesired features. In this paper, we show that conventional NP is limited by the assumption of a constant guidance scale, which may lead to highly suboptimal results, or even complete failure, due to the non-stationarity and state-dependence of the reverse process. Based on this analysis, we derive a principled technique called Dynamic Negative Guidance, which relies on a near-optimal time and state dependent modulation of the guidance without requiring additional training. Unlike NP, negative guidance requires estimating the posterior class probability during the denoising process, which is achieved with limited additional computational overhead by tracking the discrete Markov Chain during the generative process. We evaluate the performance of DNG class-removal on MNIST and CIFAR10, where we show that DNG leads to higher safety, preservation of class balance and image quality when compared with baseline methods. Furthermore, we show that it is possible to use DNG with Stable Diffusion to obtain more accurate and less invasive guidance than NP.