SecInfer: Preventing Prompt Injection via Inference-time Scaling
This addresses security vulnerabilities in LLMs for users deploying them in real-world applications, representing an incremental improvement over existing methods.
The paper tackles prompt injection attacks on Large Language Models by proposing SecInfer, a defense using inference-time scaling to generate diverse responses and aggregate them, which outperforms state-of-the-art defenses in mitigating attacks.
Prompt injection attacks pose a pervasive threat to the security of Large Language Models (LLMs). State-of-the-art prevention-based defenses typically rely on fine-tuning an LLM to enhance its security, but they achieve limited effectiveness against strong attacks. In this work, we propose \emph{SecInfer}, a novel defense against prompt injection attacks built on \emph{inference-time scaling}, an emerging paradigm that boosts LLM capability by allocating more compute resources for reasoning during inference. SecInfer consists of two key steps: \emph{system-prompt-guided sampling}, which generates multiple responses for a given input by exploring diverse reasoning paths through a varied set of system prompts, and \emph{target-task-guided aggregation}, which selects the response most likely to accomplish the intended task. Extensive experiments show that, by leveraging additional compute at inference, SecInfer effectively mitigates both existing and adaptive prompt injection attacks, outperforming state-of-the-art defenses as well as existing inference-time scaling approaches.