A Theoretical Game of Attacks via Compositional Skills
For LLM safety researchers, this work provides a theoretical foundation for understanding and improving adversarial robustness, though the practical gains are incremental.
This paper formalizes a game-theoretic framework for adversarial attacks on LLMs, deriving a theoretically optimal attack strategy and a provably optimal defense. The proposed attack outperforms existing methods across multiple LLMs and benchmarks.
As large language models grow increasingly capable, concerns about their safe deployment have intensified. While numerous alignment strategies aim to restrict harmful behavior, these defenses can still be circumvented through carefully designed adversarial prompts. In this work, we introduce a theoretical framework that formalizes a game between an attacker and a defender. Within this framework, we design a theoretical best-response attack strategy and show that it is closely related to many existing adversarial prompting methods. We further analyze the resulting game, characterize its equilibria, and reveal inherent advantages for the attacker. Drawing on our theoretical analysis, we also derive a provably optimal defense strategy. Empirically, we evaluate a practical instantiation of the theoretically optimal attack and observe stronger performance relative to existing adversarial prompting approaches in diverse settings encompassing different LLMs and benchmarks.