InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Models
This addresses a critical issue for users of large language models by providing a more robust defense against prompt injection attacks, though it is incremental as it builds on existing guardrail methods.
The paper tackles the problem of over-defense in prompt injection guardrail models, where benign inputs are falsely flagged as malicious due to trigger word bias, and introduces InjecGuard, which reduces this bias and improves accuracy by 30.8% over the best existing model.
Prompt injection attacks pose a critical threat to large language models (LLMs), enabling goal hijacking and data leakage. Prompt guard models, though effective in defense, suffer from over-defense -- falsely flagging benign inputs as malicious due to trigger word bias. To address this issue, we introduce NotInject, an evaluation dataset that systematically measures over-defense across various prompt guard models. NotInject contains 339 benign samples enriched with trigger words common in prompt injection attacks, enabling fine-grained evaluation. Our results show that state-of-the-art models suffer from over-defense issues, with accuracy dropping close to random guessing levels (60%). To mitigate this, we propose InjecGuard, a novel prompt guard model that incorporates a new training strategy, Mitigating Over-defense for Free (MOF), which significantly reduces the bias on trigger words. InjecGuard demonstrates state-of-the-art performance on diverse benchmarks including NotInject, surpassing the existing best model by 30.8%, offering a robust and open-source solution for detecting prompt injection attacks. The code and datasets are released at https://github.com/leolee99/InjecGuard.