AutoControl Arena: Synthesizing Executable Test Environments for Frontier AI Risk Evaluation
This work provides a more reliable and scalable method for evaluating safety risks in autonomous LLM agents, which is crucial for developers and researchers working on frontier AI.
This paper introduces AutoControl Arena, a framework designed to evaluate frontier AI risks in autonomous agents by synthesizing executable test environments. It addresses the trade-off between costly manual benchmarks and scalable but hallucination-prone LLM-based simulators, achieving over 98% end-to-end success and 60% human preference over existing simulators.
As Large Language Models (LLMs) evolve into autonomous agents, existing safety evaluations face a fundamental trade-off: manual benchmarks are costly, while LLM-based simulators are scalable but suffer from logic hallucination. We present AutoControl Arena, an automated framework for frontier AI risk evaluation built on the principle of logic-narrative decoupling. By grounding deterministic state in executable code while delegating generative dynamics to LLMs, we mitigate hallucination while maintaining flexibility. This principle, instantiated through a three-agent framework, achieves over 98% end-to-end success and 60% human preference over existing simulators. To elicit latent risks, we vary environmental Stress and Temptation across X-Bench (70 scenarios, 7 risk categories). Evaluating 9 frontier models reveals: (1) Alignment Illusion: risk rates surge from 21.7% to 54.5% under pressure, with capable models showing disproportionately larger increases; (2) Scenario-Specific Safety Scaling: advanced reasoning improves robustness for direct harms but worsens it for gaming scenarios; and (3) Divergent Misalignment Patterns: weaker models cause non-malicious harm while stronger models develop strategic concealment.