Open Models, Open Risks: Measuring Unsafe Generation in Text-to-Image Models In the Wild
For the AI safety community, this work provides the first large-scale empirical assessment of jailbreak risks in the real-world T2I model ecosystem, revealing that safety degradation is not universal and identifying specific high-risk models.
This paper presents a large-scale empirical study of unsafe generation in in-the-wild text-to-image models, evaluating over 200 models from Hugging Face. It introduces Advanced ASR to better measure semantically valid unsafe outputs, finding that many downstream models retain safety but some high-risk models exist, including both explicit NSFW releases and seemingly benign models with hidden unsafe behavior.
Existing safety studies on text-to-image (T2I) jailbreaks are largely conducted in controlled in-the-lab settings, typically on a small number of canonical models. As a result, the current safety status of the rapidly growing in-the-wild T2I ecosystem remains unclear. This uncertainty is amplified by two factors: existing detector-based metrics are designed for controlled evaluation, and in-the-wild risks may arise not only from adversarial prompting, but also from unsafe release practices and unsafe model derivatives. In this paper, we present a large-scale empirical study of in-the-wild T2I safety through the lens of jailbreak. We first show that detector-only jailbreak metrics substantially overestimate practical risk over in the wild due to semantic drift and generation artifacts, and we introduce Advanced ASR to better capture semantically valid and visually plausible unsafe generation. Using this refined metric, we evaluate 200+ in-the-wild T2I models from Hugging Face under three representative jailbreak attacks. Our results show that many downstream models retain a non-trivial degree of safety even without explicit post-hoc safeguards, indicating that safety degradation in the wild is neither universal nor uniform. At the same time, we identify a set of high-risk models, including explicitly NSFW-oriented releases as well as seemingly benign models whose unsafe behavior is only exposed through systematic evaluation. We further trace these models to their release context and report high-risk cases to Hugging Face.