Towards Understanding Jailbreak Attacks in LLMs: A Representation Space Analysis
This work addresses the security issue of jailbreak attacks in LLMs for AI safety researchers, but it is incremental as it builds on existing attack methods to provide new insights.
The paper tackles the problem of understanding why some jailbreak attacks succeed in making large language models output harmful content by analyzing their representation space, finding that successful attacks move harmful prompts towards harmless ones, and validating this with experiments using hidden representations to improve attack objectives.
Large language models (LLMs) are susceptible to a type of attack known as jailbreaking, which misleads LLMs to output harmful contents. Although there are diverse jailbreak attack strategies, there is no unified understanding on why some methods succeed and others fail. This paper explores the behavior of harmful and harmless prompts in the LLM's representation space to investigate the intrinsic properties of successful jailbreak attacks. We hypothesize that successful attacks share some similar properties: They are effective in moving the representation of the harmful prompt towards the direction to the harmless prompts. We leverage hidden representations into the objective of existing jailbreak attacks to move the attacks along the acceptance direction, and conduct experiments to validate the above hypothesis using the proposed objective. We hope this study provides new insights into understanding how LLMs understand harmfulness information.