Distilling Adversarial Prompts from Safety Benchmarks: Report for the
  Adversarial Nibbler Challenge
Manuel Brack Patrick Schramowski Kristian Kersting
Text-conditioned image generation models have recently achieved astonishing
image quality and alignment results. Consequently, they are employed in a
fast-growing number of applications. Since they are highly data-driven, relying
on billion-sized datasets randomly scraped from the web, they also produce
unsafe content. As a contribution to the Adversarial Nibbler challenge, we
distill a large set of over 1,000 potential adversarial inputs from existing
safety benchmarks. Our analysis of the gathered prompts and corresponding
images demonstrates the fragility of input filters and provides further
insights into systematic safety issues in current generative image models.