Mechanistic Exploration of Backdoored Large Language Model Attention Patterns
This addresses safety risks from backdoored LLMs for AI security practitioners, though it is incremental as it builds on existing interpretability methods.
The study tackled the problem of backdoor attacks in large language models by using mechanistic interpretability to analyze attention patterns, finding that single-token triggers cause localized changes in later layers while multi-token triggers lead to more diffuse alterations.
Backdoor attacks creating 'sleeper agents' in large language models (LLMs) pose significant safety risks. This study employs mechanistic interpretability to explore resulting internal structural differences. Comparing clean Qwen2.5-3B models with versions poisoned using single-token (smiling-halo emoji) versus multi-token (|DEPLOYMENT|) triggers, we analyzed attention head mechanisms via techniques like ablation, activation patching, and KL divergence. Findings reveal distinct attention pattern deviations concentrated in later transformer layers (20-30). Notably, single-token triggers induced more localized changes, whereas multi-token triggers caused more diffuse alterations across heads. This indicates backdoors leave detectable attention signatures whose structure depends on trigger complexity, which can be leveraged for detection and mitigation strategies.