Inference-Only Prompt Projection for Safe Text-to-Image Generation with TV Guarantees
For practitioners deploying T2I models, SPOT provides a training-free inference-time method to enforce safety with provable total variation guarantees, balancing safety and alignment.
The paper addresses the tension between suppressing unsafe generations and preserving benign behavior in text-to-image diffusion models. SPOT achieves 14.2% to 44.4% relative inappropriate score reductions across four datasets and three backbones while maintaining benign prompt behavior close to the reference.
Text-to-Image (T2I) diffusion models enable high quality open ended synthesis, but practical use requires suppressing unsafe generations while preserving behavior on benign prompts. We study this tension relative to the frozen generator, using its prompt conditioned distribution as the preservation reference. Since T2I safety is commonly evaluated by bounded risk scores on generated images, total variation (TV) bounds how much expected risk can change from this reference. We call this fixed reference constraint the Safety-Prompt Alignment Tradeoff (SPAT): reducing expected unsafety requires prompt conditioned distributional deviation. To make this deviation selective and adjustable, we define the tau safe set as prompts whose reference risk is at most tau, and cast intervention as projection toward nearby prompts in this set. We propose Selective Prompt prOjecTion (SPOT), an inference time framework that approximates this projection without retraining the generator or learning a category specific rewriter. SPOT uses an LLM to rank candidate rewrites and a safeguard VLM to accept generated images under the same tau. Across four datasets and three diffusion backbones, SPOT achieves relative inappropriate (IP) score reductions from 14.2% to 44.4% over strong safety alignment baselines while keeping benign prompt behavior close to the fixed reference.