OET: Optimization-based prompt injection Evaluation Toolkit
This addresses security risks for users of LLMs by providing a rigorous evaluation tool, though it is incremental as it builds on existing defense strategies.
The paper tackles the lack of a standardized framework for evaluating prompt injection attacks and defenses in LLMs by introducing OET, an optimization-based toolkit that systematically benchmarks these across diverse datasets, revealing limitations in current defenses with some models remaining susceptible even after enhancements.
Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation, enabling their widespread adoption across various domains. However, their susceptibility to prompt injection attacks poses significant security risks, as adversarial inputs can manipulate model behavior and override intended instructions. Despite numerous defense strategies, a standardized framework to rigorously evaluate their effectiveness, especially under adaptive adversarial scenarios, is lacking. To address this gap, we introduce OET, an optimization-based evaluation toolkit that systematically benchmarks prompt injection attacks and defenses across diverse datasets using an adaptive testing framework. Our toolkit features a modular workflow that facilitates adversarial string generation, dynamic attack execution, and comprehensive result analysis, offering a unified platform for assessing adversarial robustness. Crucially, the adaptive testing framework leverages optimization methods with both white-box and black-box access to generate worst-case adversarial examples, thereby enabling strict red-teaming evaluations. Extensive experiments underscore the limitations of current defense mechanisms, with some models remaining susceptible even after implementing security enhancements.