Runtime Governance for AI Agents: Policies on Paths
This addresses the challenge of balancing task success with legal and reputational risks in AI agent deployment, proposing a foundational framework for runtime governance that could impact organizations using such systems.
The paper tackles the problem of governing AI agents' non-deterministic, path-dependent behavior by formalizing runtime compliance policies as functions that map agent identity, partial execution paths, proposed actions, and organizational state to violation probabilities, arguing this is necessary for effective governance beyond static methods like prompts or access control.
AI agents -- systems that plan, reason, and act using large language models -- produce non-deterministic, path-dependent behavior that cannot be fully governed at design time, where with governed we mean striking the right balance between as high as possible successful task completion rate and the legal, data-breach, reputational and other costs associated with running agents. We argue that the execution path is the central object for effective runtime governance and formalize compliance policies as deterministic functions mapping agent identity, partial path, proposed next action, and organizational state to a policy violation probability. We show that prompt-level instructions (and "system prompts"), and static access control are special cases of this framework: the former shape the distribution over paths without actually evaluating them; the latter evaluates deterministic policies that ignore the path (i.e., these can only account for a specific subset of all possible paths). In our view, runtime evaluation is the general case, and it is necessary for any path-dependent policy. We develop the formal framework for analyzing AI agent governance, present concrete policy examples (inspired by the AI act), discuss a reference implementation, and identify open problems including risk calibration and the limits of enforced compliance.