Evaluating using Mock Tool Calls to Quarantine Untrusted Prompt Inputs
This research addresses a critical security vulnerability for systems using large language models that process untrusted inputs, impacting developers and users by revealing a counter-intuitive failure in a proposed mitigation strategy.
This paper investigates the effectiveness of wrapping untrusted prompt inputs in mock tool calls as a quarantine mechanism for large language models. The study found that this method does not broadly improve robustness; specifically, it increased attack success rates on a binary evaluation task (GSM8K grading) and showed smaller, model-dependent effects on scalar and pairwise tasks, with no model reliably benefiting.
Large language models must frequently process untrusted inputs, such as judging an answer from another model or running tasks like spam and harm classifiers while under adversarial pressure. These inputs are often string-formatted directly into a prompt template, leaving systems fragile to manipulation. Current LLM specs from major providers like OpenAI distinguish trustworthiness along an Instruction Hierarchy, from System messages (most trusted) to Tool Results (least trusted). A possible natural mitigation is to wrap untrusted content in a mock tool call as a quarantine. We explore this hypothesis with an automated redteaming search over static attack strings across seven models and three LLM-as-a-Judge tasks. Counter to our hypothesis, tool-wrapping does not broadly improve robustness. On a binary evaluation task (GSM8K grading) it typically increases attack success rates, an apparent inversion of the instruction hierarchy. On scalar and pairwise tasks the effect is smaller and model-dependent, with no tested model reliably helped, and several showing inversion. We recommend evaluating this limitation in deployed systems, and longer-term, pursuing stronger Instruction Hierarchy training or new untrusted-input primitives.