ContextCov: Deriving and Enforcing Executable Constraints from Agent Instruction Files
This addresses the issue of technical debt from silent violations in autonomous software engineering for developers using LLM agents, representing a novel method for a known bottleneck.
The paper tackles the problem of LLM agents deviating from natural language instructions, termed Context Drift, by introducing ContextCov, a framework that transforms these instructions into executable guardrails, achieving over 46,000 checks with 99.997% syntax validity in evaluations on 723 repositories.
As Large Language Model (LLM) agents increasingly execute complex, autonomous software engineering tasks, developers rely on natural language Agent Instructions (e.g., AGENTS.md) to enforce project-specific coding conventions, tooling, and architectural boundaries. However, these instructions are passive text. Agents frequently deviate from them due to context limitations or conflicting legacy code, a phenomenon we term Context Drift. Because agents operate without real-time human supervision, these silent violations rapidly compound into technical debt. To bridge this gap, we introduce ContextCov, a framework that transforms passive Agent Instructions into active, executable guardrails. ContextCov extracts natural language constraints and synthesizes enforcement checks across three domains: static AST analysis for code patterns, runtime shell shims that intercept prohibited commands, and architectural validators for structural and semantic constraints. Evaluations on 723 open-source repositories demonstrate that ContextCov successfully extracts over 46,000 executable checks with 99.997% syntax validity, providing a necessary automated compliance layer for safe, agent-driven development. Source code and evaluation results are available at https://anonymous.4open.science/r/ContextCov-4510/.