Adversarial Suffix Filtering: a Defense Pipeline for LLMs
This addresses security and trustworthiness issues for LLMs deployed in autonomous systems and public-facing environments, representing a strong specific gain in defense capabilities.
The paper tackles the problem of jailbreak vulnerabilities in LLMs, particularly from adversarial suffix attacks, by introducing Adversarial Suffix Filtering (ASF), a lightweight model-agnostic defense pipeline that reduces attack efficacy to below 4% while minimally impacting non-adversarial performance.
Large Language Models (LLMs) are increasingly embedded in autonomous systems and public-facing environments, yet they remain susceptible to jailbreak vulnerabilities that may undermine their security and trustworthiness. Adversarial suffixes are considered to be the current state-of-the-art jailbreak, consistently outperforming simpler methods and frequently succeeding even in black-box settings. Existing defenses rely on access to the internal architecture of models limiting diverse deployment, increase memory and computation footprints dramatically, or can be bypassed with simple prompt engineering methods. We introduce $\textbf{Adversarial Suffix Filtering}$ (ASF), a lightweight novel model-agnostic defensive pipeline designed to protect LLMs against adversarial suffix attacks. ASF functions as an input preprocessor and sanitizer that detects and filters adversarially crafted suffixes in prompts, effectively neutralizing malicious injections. We demonstrate that ASF provides comprehensive defense capabilities across both black-box and white-box attack settings, reducing the attack efficacy of state-of-the-art adversarial suffix generation methods to below 4%, while only minimally affecting the target model's capabilities in non-adversarial scenarios.