When Prompt Optimization Becomes Jailbreaking: Adaptive Red-Teaming of Large Language Models
This work addresses the problem of underestimating residual safety risks in LLMs for high-stakes applications by highlighting the need for adaptive red-teaming in safety evaluations.
The study tackled the vulnerability of large language models to automated adversarial prompt refinement by repurposing black-box prompt optimization techniques to systematically search for safety failures, resulting in a substantial reduction in effective safety safeguards, such as increasing the average danger score of Qwen 3 8B from 0.09 to 0.79 after optimization.
Large Language Models (LLMs) are increasingly integrated into high-stakes applications, making robust safety guarantees a central practical and commercial concern. Existing safety evaluations predominantly rely on fixed collections of harmful prompts, implicitly assuming non-adaptive adversaries and thereby overlooking realistic attack scenarios in which inputs are iteratively refined to evade safeguards. In this work, we examine the vulnerability of contemporary language models to automated, adversarial prompt refinement. We repurpose black-box prompt optimization techniques, originally designed to improve performance on benign tasks, to systematically search for safety failures. Using DSPy, we apply three such optimizers to prompts drawn from HarmfulQA and JailbreakBench, explicitly optimizing toward a continuous danger score in the range 0 to 1 provided by an independent evaluator model (GPT-5.1). Our results demonstrate a substantial reduction in effective safety safeguards, with the effects being especially pronounced for open-source small language models. For example, the average danger score of Qwen 3 8B increases from 0.09 in its baseline setting to 0.79 after optimization. These findings suggest that static benchmarks may underestimate residual risk, indicating that automated, adaptive red-teaming is a necessary component of robust safety evaluation.