Sparse Autoencoders are Capable LLM Jailbreak Mitigators
This addresses the problem of jailbreak attacks for users of aligned LLMs, offering a practical defense without task-specific training, though it is incremental as it builds on existing sparse autoencoder methods.
The paper tackles jailbreak attacks on large language models by proposing Context-Conditioned Delta Steering (CC-Delta), a defense using sparse autoencoders to identify and steer harmful features, achieving comparable or better safety-utility tradeoffs than dense-space baselines across four models and twelve attacks, with clear outperformance against out-of-distribution attacks.
Jailbreak attacks remain a persistent threat to large language model safety. We propose Context-Conditioned Delta Steering (CC-Delta), an SAE-based defense that identifies jailbreak-relevant sparse features by comparing token-level representations of the same harmful request with and without jailbreak context. Using paired harmful/jailbreak prompts, CC-Delta selects features via statistical testing and applies inference-time mean-shift steering in SAE latent space. Across four aligned instruction-tuned models and twelve jailbreak attacks, CC-Delta achieves comparable or better safety-utility tradeoffs than baseline defenses operating in dense latent space. In particular, our method clearly outperforms dense mean-shift steering on all four models, and particularly against out-of-distribution attacks, showing that steering in sparse SAE feature space offers advantages over steering in dense activation space for jailbreak mitigation. Our results suggest off-the-shelf SAEs trained for interpretability can be repurposed as practical jailbreak defenses without task-specific training.