OpenSec: Measuring Incident Response Agent Calibration Under Adversarial Evidence
This addresses a calibration failure mode in defensive incident response agents for cybersecurity, which is incremental as it focuses on a specific hidden issue in existing benchmarks.
The paper tackled the problem of measuring incident response agent calibration under adversarial evidence, finding that frontier models like GPT-5.2, Gemini 3, and DeepSeek executed containment in 100% of episodes with 90-97% false positive rates, while Claude Sonnet 4.5 showed partial calibration with 85% containment and 72% false positives.
As large language models improve, so do their offensive applications: frontier agents now generate working exploits for under $50 in compute (Heelan, 2026). Defensive incident response (IR) agents must keep pace, but existing benchmarks conflate action execution with correct execution, hiding calibration failures when agents process adversarial evidence. We introduce OpenSec, a dual-control reinforcement learning environment that evaluates IR agents under realistic prompt injection scenarios. Unlike static capability benchmarks, OpenSec scores world-state-changing containment actions under adversarial evidence via execution-based metrics: time-to-first-containment (TTFC), blast radius (false positives per episode), and injection violation rates. Evaluating four frontier models on 40 standard-tier episodes, we find consistent over-triggering in this setting: GPT-5.2, Gemini 3, and DeepSeek execute containment in 100% of episodes with 90-97% false positive rates. Claude Sonnet 4.5 shows partial calibration (85% containment, 72% FP), demonstrating that OpenSec surfaces a calibration failure mode hidden by aggregate success metrics. Code available at https://github.com/jbarnes850/opensec-env.