Evaluating Control Protocols for Untrusted AI Agents
This work addresses safety risks from untrusted AI agents for AI deployment, but it is incremental as it focuses on evaluating existing protocols rather than proposing new ones.
The paper tackles the problem of evaluating safety protocols for untrusted AI agents by systematically testing control methods like resampling and deferring on critical actions in diverse environments, finding that resampling increases safety from 50% to 96% but can drop to 17% under adaptive attacks, while deferring on critical actions remains robust.
As AI systems become more capable and widely deployed as agents, ensuring their safe operation becomes critical. AI control offers one approach to mitigating the risk from untrusted AI agents by monitoring their actions and intervening or auditing when necessary. Evaluating the safety of these protocols requires understanding both their effectiveness against current attacks and their robustness to adaptive adversaries. In this work, we systematically evaluate a range of control protocols in SHADE-Arena, a dataset of diverse agentic environments. First, we evaluate blue team protocols, including deferral to trusted models, resampling, and deferring on critical actions, against a default attack policy. We find that resampling for incrimination and deferring on critical actions perform best, increasing safety from 50% to 96%. We then iterate on red team strategies against these protocols and find that attack policies with additional affordances, such as knowledge of when resampling occurs or the ability to simulate monitors, can substantially improve attack success rates against our resampling strategy, decreasing safety to 17%. However, deferring on critical actions is highly robust to even our strongest red team strategies, demonstrating the importance of denying attack policies access to protocol internals.