Rule Encoding and Compliance in Large Language Models: An Information-Theoretic Analysis
This work addresses the challenge of protecting LLM-based agents against prompt injection attacks while maintaining compliance in evolving domains, representing a novel method for a known bottleneck rather than a foundational advancement.
The paper tackles the problem of ensuring rule compliance in large language models (LLMs) for safety-critical agents by analyzing how rule encodings affect attention mechanisms, revealing a trade-off between anchor redundancy and attention entropy that improves pointer fidelity and increases the asymptotic probability of compliant outputs.
The design of safety-critical agents based on large language models (LLMs) requires more than simple prompt engineering. This paper presents a comprehensive information-theoretic analysis of how rule encodings in system prompts influence attention mechanisms and compliance behaviour. We demonstrate that rule formats with low syntactic entropy and highly concentrated anchors reduce attention entropy and improve pointer fidelity, but reveal a fundamental trade-off between anchor redundancy and attention entropy that previous work failed to recognize. Through formal analysis of multiple attention architectures including causal, bidirectional, local sparse, kernelized, and cross-attention mechanisms, we establish bounds on pointer fidelity and show how anchor placement strategies must account for competing fidelity and entropy objectives. Combining these insights with a dynamic rule verification architecture, we provide a formal proof that hot reloading of verified rule sets increases the asymptotic probability of compliant outputs. These findings underscore the necessity of principled anchor design and dual enforcement mechanisms to protect LLM-based agents against prompt injection attacks while maintaining compliance in evolving domains.