Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations
This addresses safety risks in LLMs for users and developers, but it is incremental as it builds on existing in-context learning methods.
The paper tackles the problem of safety alignment in large language models (LLMs) by exploring how in-context learning can be used to both jailbreak and defend against harmful content generation, showing that a few demonstrations can significantly increase or decrease jailbreaking success rates.
Large Language Models (LLMs) have shown remarkable success in various tasks, yet their safety and the risk of generating harmful content remain pressing concerns. In this paper, we delve into the potential of In-Context Learning (ICL) to modulate the alignment of LLMs. Specifically, we propose the In-Context Attack (ICA) which employs harmful demonstrations to subvert LLMs, and the In-Context Defense (ICD) which bolsters model resilience through examples that demonstrate refusal to produce harmful responses. We offer theoretical insights to elucidate how a limited set of in-context demonstrations can pivotally influence the safety alignment of LLMs. Through extensive experiments, we demonstrate the efficacy of ICA and ICD in respectively elevating and mitigating the success rates of jailbreaking prompts. Our findings illuminate the profound influence of ICL on LLM behavior, opening new avenues for improving the safety of LLMs.