A Dynamic Stackelberg Game Framework for Agentic AI Defense Against LLM Jailbreaking
This addresses the security challenge of jailbreaking for LLMs in critical applications, representing a novel method for a known bottleneck rather than a foundational breakthrough.
The paper tackles the problem of LLM jailbreaking by adversaries by proposing a dynamic Stackelberg game framework to model attacker-defender interactions, resulting in the development of the 'Purple Agent' that uses adversarial exploration and proactive intervention to prevent harmful outputs.
As large language models (LLMs) are increasingly deployed in critical applications, the challenge of jailbreaking, where adversaries manipulate the models to bypass safety mechanisms, has become a significant concern. This paper presents a dynamic Stackelberg game framework to model the interactions between attackers and defenders in the context of LLM jailbreaking. The framework treats the prompt-response dynamics as a sequential extensive-form game, where the defender, as the leader, commits to a strategy while anticipating the attacker's optimal responses. We propose a novel agentic AI solution, the "Purple Agent," which integrates adversarial exploration and defensive strategies using Rapidly-exploring Random Trees (RRT). The Purple Agent actively simulates potential attack trajectories and intervenes proactively to prevent harmful outputs. This approach offers a principled method for analyzing adversarial dynamics and provides a foundation for mitigating the risk of jailbreaking.