Automated Framework to Evaluate and Harden LLM System Instructions against Encoding Attacks
This addresses a critical security risk for LLM applications, as it reveals vulnerabilities in refusal-based safety mechanisms, though it is incremental by building on existing attack methods.
The paper tackled the problem of system instruction leakage in LLMs by introducing an automated framework to test confidentiality against encoding attacks, finding high attack success rates (>0.7) across models and instructions, and demonstrating a mitigation strategy that reduces these rates without retraining.
System Instructions in Large Language Models (LLMs) are commonly used to enforce safety policies, define agent behavior, and protect sensitive operational context in agentic AI applications. These instructions may contain sensitive information such as API credentials, internal policies, and privileged workflow definitions, making system instruction leakage a critical security risk highlighted in the OWASP Top 10 for LLM Applications. Without incurring the overhead costs of reasoning models, many LLM applications rely on refusal-based instructions that block direct requests for system instructions, implicitly assuming that prohibited information can only be extracted through explicit queries. We introduce an automated evaluation framework that tests whether system instructions remain confidential when extraction requests are re-framed as encoding or structured output tasks. Across four common models and 46 verified system instructions, we observe high attack success rates (> 0.7) for structured serialization where models refuse direct extraction requests but disclose protected content in the requested serialization formats. We further demonstrate a mitigation strategy based on one-shot instruction reshaping using a Chain-of-Thought reasoning model, indicating that even subtle changes in wording and structure of system instructions can significantly reduce attack success rate without requiring model retraining.