Tempest: Autonomous Multi-Turn Jailbreaking of Large Language Models with Tree Search
This work addresses the vulnerability of LLMs to multi-turn attacks, highlighting a critical safety issue for AI developers and users, though it is incremental in building on existing jailbreaking methods.
The paper tackles the problem of jailbreaking large language models by introducing Tempest, a multi-turn adversarial framework that uses tree search to gradually erode model safety, achieving a 100% success rate on GPT-3.5-turbo and 97% on GPT-4 in evaluations.
We introduce Tempest, a multi-turn adversarial framework that models the gradual erosion of Large Language Model (LLM) safety through a tree search perspective. Unlike single-turn jailbreaks that rely on one meticulously engineered prompt, Tempest expands the conversation at each turn in a breadth-first fashion, branching out multiple adversarial prompts that exploit partial compliance from previous responses. By tracking these incremental policy leaks and re-injecting them into subsequent queries, Tempest reveals how minor concessions can accumulate into fully disallowed outputs. Evaluations on the JailbreakBench dataset show that Tempest achieves a 100% success rate on GPT-3.5-turbo and 97% on GPT-4 in a single multi-turn run, using fewer queries than baselines such as Crescendo or GOAT. This tree search methodology offers an in-depth view of how model safeguards degrade over successive dialogue turns, underscoring the urgency of robust multi-turn testing procedures for language models.