Exploiting Prefix-Tree in Structured Output Interfaces for Enhancing Jailbreak Attacking
This work addresses security vulnerabilities in LLMs for providers and users, though it is incremental as it builds on existing jailbreak techniques.
The paper tackles the problem of jailbreak attacks on Large Language Models (LLMs) by targeting structured output interfaces, introducing a black-box attack framework called AttackPrefixTree (APT) that achieves a higher attack success rate than existing methods.
The rise of Large Language Models (LLMs) has led to significant applications but also introduced serious security threats, particularly from jailbreak attacks that manipulate output generation. These attacks utilize prompt engineering and logit manipulation to steer models toward harmful content, prompting LLM providers to implement filtering and safety alignment strategies. We investigate LLMs' safety mechanisms and their recent applications, revealing a new threat model targeting structured output interfaces, which enable attackers to manipulate the inner logit during LLM generation, requiring only API access permissions. To demonstrate this threat model, we introduce a black-box attack framework called AttackPrefixTree (APT). APT exploits structured output interfaces to dynamically construct attack patterns. By leveraging prefixes of models' safety refusal response and latent harmful outputs, APT effectively bypasses safety measures. Experiments on benchmark datasets indicate that this approach achieves higher attack success rate than existing methods. This work highlights the urgent need for LLM providers to enhance security protocols to address vulnerabilities arising from the interaction between safety patterns and structured outputs.