DROJ: A Prompt-Driven Attack against Large Language Models
This addresses security vulnerabilities in LLMs for users and developers, though it is incremental as it builds on existing jailbreak techniques.
The paper tackles the problem of adversarial jailbreak attacks on large language models by introducing DROJ, a method that optimizes prompts at the embedding level to shift harmful query representations, achieving a 100% keyword-based Attack Success Rate on LLaMA-2-7b-chat while addressing issues like repetitive responses with a helpfulness prompt.
Large Language Models (LLMs) have demonstrated exceptional capabilities across various natural language processing tasks. Due to their training on internet-sourced datasets, LLMs can sometimes generate objectionable content, necessitating extensive alignment with human feedback to avoid such outputs. Despite massive alignment efforts, LLMs remain susceptible to adversarial jailbreak attacks, which usually are manipulated prompts designed to circumvent safety mechanisms and elicit harmful responses. Here, we introduce a novel approach, Directed Rrepresentation Optimization Jailbreak (DROJ), which optimizes jailbreak prompts at the embedding level to shift the hidden representations of harmful queries towards directions that are more likely to elicit affirmative responses from the model. Our evaluations on LLaMA-2-7b-chat model show that DROJ achieves a 100\% keyword-based Attack Success Rate (ASR), effectively preventing direct refusals. However, the model occasionally produces repetitive and non-informative responses. To mitigate this, we introduce a helpfulness system prompt that enhances the utility of the model's responses. Our code is available at https://github.com/Leon-Leyang/LLM-Safeguard.