Distilling Adversarial Prompts from Safety Benchmarks: Report for the Adversarial Nibbler Challenge
This work addresses safety vulnerabilities in generative image models, which is crucial for real-world applications, but it is incremental as it builds on existing benchmarks.
The authors tackled the problem of unsafe content generation by text-conditioned image models by distilling over 1,000 potential adversarial inputs from safety benchmarks, revealing the fragility of input filters and systematic safety issues.
Text-conditioned image generation models have recently achieved astonishing image quality and alignment results. Consequently, they are employed in a fast-growing number of applications. Since they are highly data-driven, relying on billion-sized datasets randomly scraped from the web, they also produce unsafe content. As a contribution to the Adversarial Nibbler challenge, we distill a large set of over 1,000 potential adversarial inputs from existing safety benchmarks. Our analysis of the gathered prompts and corresponding images demonstrates the fragility of input filters and provides further insights into systematic safety issues in current generative image models.