Persona Attack: Incremental Memory Injection Jailbreak Attack against Large Language Models
It demonstrates a new vulnerability in LLMs that can bypass safety training, posing a security risk for deployed conversational AI systems.
The paper proposes Persona Attack, a jailbreak method that incrementally injects instructions into an LLM's memory, achieving up to 95% attack success rate by exploiting the model's tendency to prioritize accumulated instructions over safety alignment.
As Large Language Models evolve for user convenience, vulnerability to jailbreak attacks continues to be reported despite ongoing efforts in safety training. Traditional jailbreak techniques typically focus on a single prompt injection, neglecting the models' ability to remember the flow of conversation and the user's instructions. In this paper, we propose Persona Attack, a memory injection based jailbreak method that manipulates the model's context window through a step by step approach. Experimental results from applying Persona Attack to several widely used LLMs reveal that, as injections accumulate in memory, models increasingly prioritize these instructions over their internal safety alignment mechanisms. Furthermore, our experiments empirically demonstrate that the attack success rate varies not only according to the memory implementation of the model, but also combinations of instructions and can reach 95% under specific instruction configurations.