Negative Token Merging: Image-based Adversarial Feature Guidance
This addresses the problem of controlling complex visual concepts in diffusion models for users in creative and commercial applications, though it is an incremental improvement over existing text-based adversarial guidance.
The paper tackles the problem of insufficient adversarial guidance in diffusion models when using only text prompts, by introducing a training-free method called negative token merging (NegToMe) that uses visual features from reference images to steer generation away from undesired concepts. The result is a 34.57% reduction in visual similarity to copyrighted content and enhanced output diversity with only marginally higher inference time.
Text-based adversarial guidance using a negative prompt has emerged as a widely adopted approach to steer diffusion models away from producing undesired concepts. While useful, performing adversarial guidance using text alone can be insufficient to capture complex visual concepts or avoid specific visual elements like copyrighted characters. In this paper, for the first time we explore an alternate modality in this direction by performing adversarial guidance directly using visual features from a reference image or other images in a batch. We introduce negative token merging (NegToMe), a simple but effective training-free approach which performs adversarial guidance through images by selectively pushing apart matching visual features between reference and generated images during the reverse diffusion process. By simply adjusting the used reference, NegToMe enables a diverse range of applications. Notably, when using other images in same batch as reference, we find that NegToMe significantly enhances output diversity (e.g., racial, gender, visual) by guiding features of each image away from others. Similarly, when used w.r.t. copyrighted reference images, NegToMe reduces visual similarity to copyrighted content by 34.57%. NegToMe is simple to implement using just few-lines of code, uses only marginally higher (<4%) inference time and is compatible with different diffusion architectures, including those like Flux, which don't natively support the use of a negative prompt. Code is available at https://negtome.github.io