Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?
This addresses a critical safety vulnerability in LLMs for practical applications, though it is incremental as it defines and benchmarks an existing problem.
The authors tackled the lack of safety features in instruction-tuned LLMs by introducing a formal measure and dataset to quantify instruction-data separation, finding that all tested models fail to achieve high separation and mitigation techniques are ineffective.
Instruction-tuned Large Language Models (LLMs) show impressive results in numerous practical applications, but they lack essential safety features that are common in other areas of computer science, particularly an explicit separation of instructions and data. This makes them vulnerable to manipulations such as indirect prompt injections and generally unsuitable for safety-critical tasks. Surprisingly, there is currently no established definition or benchmark to quantify this phenomenon. In this work, we close this gap by introducing a formal measure for instruction-data separation and an empirical variant that is calculable from a model's outputs. We also present a new dataset, SEP, that allows estimating the measure for real-world models. Our results on various LLMs show that the problem of instruction-data separation is real: all models fail to achieve high separation, and canonical mitigation techniques, such as prompt engineering and fine-tuning, either fail to substantially improve separation or reduce model utility. The source code and SEP dataset are openly accessible at https://github.com/egozverev/Shold-It-Be-Executed-Or-Processed.