Jailbreak Scaling Laws for Large Language Models: Polynomial-Exponential Crossover
This work addresses the vulnerability of large language models to jailbreaking attacks, which is a critical safety issue for AI deployment, though it is incremental in building on existing adversarial attack research.
The paper tackles the problem of adversarial prompt-injection attacks on safety-aligned large language models, finding that such attacks can amplify attack success rates from polynomial to exponential growth with inference-time samples, as confirmed empirically and explained through a theoretical spin-glass model.
Adversarial attacks can reliably steer safety-aligned large language models toward unsafe behavior. Empirically, we find that adversarial prompt-injection attacks can amplify attack success rate from the slow polynomial growth observed without injection to exponential growth with the number of inference-time samples. To explain this phenomenon, we propose a theoretical generative model of proxy language in terms of a spin-glass system operating in a replica-symmetry-breaking regime, where generations are drawn from the associated Gibbs measure and a subset of low-energy, size-biased clusters is designated unsafe. Within this framework, we analyze prompt injection-based jailbreaking. Short injected prompts correspond to a weak magnetic field aligned towards unsafe cluster centers and yield a power-law scaling of attack success rate with the number of inference-time samples, while long injected prompts, i.e., strong magnetic field, yield exponential scaling. We derive these behaviors analytically and confirm them empirically on large language models. This transition between two regimes is due to the appearance of an ordered phase in the spin chain under a strong magnetic field, which suggests that the injected jailbreak prompt enhances adversarial order in the language model.