Inference-Time Vulnerability Beyond Shallow Safety: Alignment Along Generation Trajectories
For developers of safety-aligned LLMs, the paper identifies a broader vulnerability and proposes a training method that improves robustness, though the gains are incremental over existing alignment techniques.
The paper reveals that safety-aligned LLMs are vulnerable to short token injections at any generation step, not just early tokens, and that internal refusal directions do not predict robustness. By aligning models on perturbed generation trajectories, they improve robustness to mid-sequence attacks and generalize to early-token attacks.
Safety-aligned Large Language Models (LLMs) remain vulnerable to interventions during inference that redirect generation toward harmful outputs. Recent work attributes this to shallow safety, where alignment concentrates in the first few output tokens. We show that shallow safety is a special case of a broader inference-time vulnerability, in which short token injections at any generation step can substantially alter subsequent safety behavior. We also find that a model's alignment with refusal directions in its hidden states does not predict its robustness to such injection, revealing that internal state alone does not determine generation behavior under perturbation. To address this, we align models directly on generation trajectories constructed by simulating mid-sequence perturbation, and show that this improves robustness to mid-sequence injection and generalizes to attacks that exploit early-token generation. Our work argues that robust safety alignment requires training on the generation process itself, not only its outputs.