DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks
This addresses a critical security vulnerability for users of LLM-integrated applications and agents, though it is an incremental improvement over existing detection methods.
The paper tackles the problem of detecting prompt injection attacks in LLM-integrated applications by proposing DataSentinel, a game-theoretic method that fine-tunes an LLM to detect inputs contaminated with strategically adapted injected prompts, achieving effective detection on multiple benchmark datasets and LLMs.
LLM-integrated applications and agents are vulnerable to prompt injection attacks, where an attacker injects prompts into their inputs to induce attacker-desired outputs. A detection method aims to determine whether a given input is contaminated by an injected prompt. However, existing detection methods have limited effectiveness against state-of-the-art attacks, let alone adaptive ones. In this work, we propose DataSentinel, a game-theoretic method to detect prompt injection attacks. Specifically, DataSentinel fine-tunes an LLM to detect inputs contaminated with injected prompts that are strategically adapted to evade detection. We formulate this as a minimax optimization problem, with the objective of fine-tuning the LLM to detect strong adaptive attacks. Furthermore, we propose a gradient-based method to solve the minimax optimization problem by alternating between the inner max and outer min problems. Our evaluation results on multiple benchmark datasets and LLMs show that DataSentinel effectively detects both existing and adaptive prompt injection attacks.