Efficient and Sound Probabilistic Verification for AI Agents
For AI security researchers and developers, this provides a method to enforce probabilistic policies without independence assumptions, addressing a critical gap in runtime monitoring of AI agents.
The paper introduces a sound and efficient probabilistic verification framework for AI agents that enforces security policies with probabilistic predicates, using distributionally robust optimization to compute sound upper bounds on violation probability. On standard benchmarks, it outperforms prior art and improves the security-utility trade-off while ensuring rigorous bounds.
Securing AI agents that operate in complex digital environments has become a critical need, and runtime monitoring approaches that formulate and enforce policies expressed in a formal language like Datalog offer a promising solution. However, existing approaches are restricted to deterministic policies. In many practical applications of AI agents, there is a need to enforce security policies in the face of ambiguity, leading to probabilistic predicates or state transitions (for example, a declassifier or Personally Identifiable Information (PII) detector that has some failure probability on each invocation). Furthermore, in many such applications, one cannot easily make the independence assumptions necessary to invoke prior work on probabilistic inference in Datalog. We address this by introducing a sound and efficient framework for such verification based on distributionally robust optimization, computing sound upper bounds on the probability of policy violation regardless of possible correlations between predicates. On standard benchmarks for terminal and tool calling agents, we demonstrate that our approach outperforms prior art and improves the security-utility trade-off while ensuring rigorous bounds on the probability of policy violation.