GRP-Obliteration: Unaligning LLMs With a Single Unlabeled Prompt
This addresses a critical security issue for AI developers and users by exposing a practical weakness in safety mechanisms, though it is incremental as it builds on existing unalignment methods.
The paper tackles the problem of safety alignment in large language models being vulnerable to post-deployment unalignment, and introduces GRP-Obliteration, a method that uses a single unlabeled prompt to reliably remove safety constraints while largely preserving utility, achieving stronger unalignment than existing state-of-the-art techniques across multiple models and benchmarks.
Safety alignment is only as robust as its weakest failure mode. Despite extensive work on safety post-training, it has been shown that models can be readily unaligned through post-deployment fine-tuning. However, these methods often require extensive data curation and degrade model utility. In this work, we extend the practical limits of unalignment by introducing GRP-Obliteration (GRP-Oblit), a method that uses Group Relative Policy Optimization (GRPO) to directly remove safety constraints from target models. We show that a single unlabeled prompt is sufficient to reliably unalign safety-aligned models while largely preserving their utility, and that GRP-Oblit achieves stronger unalignment on average than existing state-of-the-art techniques. Moreover, GRP-Oblit generalizes beyond language models and can also unalign diffusion-based image generation systems. We evaluate GRP-Oblit on six utility benchmarks and five safety benchmarks across fifteen 7-20B parameter models, spanning instruct and reasoning models, as well as dense and MoE architectures. The evaluated model families include GPT-OSS, distilled DeepSeek, Gemma, Llama, Ministral, and Qwen.