Prompt Injection as Role Confusion
This addresses a critical security problem for users of language models by revealing a fundamental gap in how authority is assigned, with broad implications for AI safety.
The paper tackled the vulnerability of language models to prompt injection attacks by identifying role confusion as the underlying mechanism, where models assign authority based on text style rather than source, and demonstrated this with spoofed reasoning achieving average success rates of 60% on StrongREJECT and 61% on agent exfiltration across various models.
Language models remain vulnerable to prompt injection attacks despite extensive safety training. We trace this failure to role confusion: models infer roles from how text is written, not where it comes from. We design novel role probes to capture how models internally identify "who is speaking." These reveal why prompt injection works: untrusted text that imitates a role inherits that role's authority. We test this insight by injecting spoofed reasoning into user prompts and tool outputs, achieving average success rates of 60% on StrongREJECT and 61% on agent exfiltration, across multiple open- and closed-weight models with near-zero baselines. Strikingly, the degree of internal role confusion strongly predicts attack success before generation begins. Our findings reveal a fundamental gap: security is defined at the interface but authority is assigned in latent space. More broadly, we introduce a unifying, mechanistic framework for prompt injection, demonstrating that diverse prompt-injection attacks exploit the same underlying role-confusion mechanism.