Ignore Previous Prompt: Attack Techniques For Language Models
This addresses security risks for users and developers of LLMs in customer-facing applications, though it is incremental as it builds on existing adversarial attack research.
The paper tackles the problem of vulnerabilities in large language models (LLMs) like GPT-3 from malicious user interactions, demonstrating that simple handcrafted inputs can easily misalign the model through attacks such as goal hijacking and prompt leaking, creating long-tail risks.
Transformer-based large language models (LLMs) provide a powerful foundation for natural language tasks in large-scale customer-facing applications. However, studies that explore their vulnerabilities emerging from malicious user interaction are scarce. By proposing PromptInject, a prosaic alignment framework for mask-based iterative adversarial prompt composition, we examine how GPT-3, the most widely deployed language model in production, can be easily misaligned by simple handcrafted inputs. In particular, we investigate two types of attacks -- goal hijacking and prompt leaking -- and demonstrate that even low-aptitude, but sufficiently ill-intentioned agents, can easily exploit GPT-3's stochastic nature, creating long-tail risks. The code for PromptInject is available at https://github.com/agencyenterprise/PromptInject.