Adversarial Attacks on Image Generation With Made-Up Words
This work addresses vulnerabilities in text-guided image generation models for content moderation systems, posing risks for generating offensive content.
The paper tackles the problem of generating images using adversarially designed nonce words to evoke specific visual concepts, introducing macaronic and evocative prompting methods that can circumvent content moderation and generate harmful images.
Text-guided image generation models can be prompted to generate images using nonce words adversarially designed to robustly evoke specific visual concepts. Two approaches for such generation are introduced: macaronic prompting, which involves designing cryptic hybrid words by concatenating subword units from different languages; and evocative prompting, which involves designing nonce words whose broad morphological features are similar enough to that of existing words to trigger robust visual associations. The two methods can also be combined to generate images associated with more specific visual concepts. The implications of these techniques for the circumvention of existing approaches to content moderation, and particularly the generation of offensive or harmful images, are discussed.