Latent Guard: a Safety Framework for Text-to-image Generation
This addresses safety concerns for users of text-to-image models by providing a more robust alternative to existing methods, though it appears incremental as it builds on blacklist-based approaches.
The paper tackles the problem of preventing misuse in text-to-image generation by proposing Latent Guard, a safety framework that learns a latent space to detect harmful concepts in input text embeddings, achieving effectiveness verified on three datasets against four baselines.
With the ability to generate high-quality images, text-to-image (T2I) models can be exploited for creating inappropriate content. To prevent misuse, existing safety measures are either based on text blacklists, which can be easily circumvented, or harmful content classification, requiring large datasets for training and offering low flexibility. Hence, we propose Latent Guard, a framework designed to improve safety measures in text-to-image generation. Inspired by blacklist-based approaches, Latent Guard learns a latent space on top of the T2I model's text encoder, where it is possible to check the presence of harmful concepts in the input text embeddings. Our proposed framework is composed of a data generation pipeline specific to the task using large language models, ad-hoc architectural components, and a contrastive learning strategy to benefit from the generated data. The effectiveness of our method is verified on three datasets and against four baselines. Code and data will be shared at https://latentguard.github.io/.