How Not to Detect Prompt Injections with an LLM
This work exposes a critical security flaw in widely used LLM defenses, posing a significant risk for applications relying on these systems.
The paper tackles the vulnerability of LLM-integrated applications to prompt injection attacks, showing that a recent defense method (known-answer detection) is fundamentally flawed and can be evaded with detection rates as low as 1.5% while achieving malicious success rates up to 88%.
LLM-integrated applications and agents are vulnerable to prompt injection attacks, in which adversaries embed malicious instructions within seemingly benign user inputs to manipulate the LLM's intended behavior. Recent defenses based on $\textit{known-answer detection}$ (KAD) have achieved near-perfect performance by using an LLM to classify inputs as clean or contaminated. In this work, we formally characterize the KAD framework and uncover a structural vulnerability in its design that invalidates its core security premise. We design a methodical adaptive attack, $\textit{DataFlip}$, to exploit this fundamental weakness. It consistently evades KAD defenses with detection rates as low as $1.5\%$ while reliably inducing malicious behavior with success rates of up to $88\%$, without needing white-box access to the LLM or any optimization procedures.