SIR-Bench: Evaluating Investigation Depth in Security Incident Response Agents
Provides a standardized evaluation framework for security incident response agents, addressing the need to distinguish genuine investigation from superficial alert handling.
SIR-Bench is a benchmark of 794 test cases for evaluating autonomous security incident response agents, measuring genuine forensic investigation beyond alert parroting. The SIR agent achieved 97.1% true positive detection, 73.4% false positive rejection, and 5.67 novel key findings per case.
We present SIR-Bench, a benchmark of 794 test cases for evaluating autonomous security incident response agents that distinguishes genuine forensic investigation from alert parroting. Derived from 129 anonymized incident patterns with expert-validated ground truth, SIR-Bench measures not only whether agents reach correct triage decisions, but whether they discover novel evidence through active investigation. To construct SIR-Bench, we develop Once Upon A Threat (OUAT), a framework that replays real incident patterns in controlled cloud environments, producing authentic telemetry with measurable investigation outcomes. Our evaluation methodology introduces three complementary metrics: triage accuracy (M1), novel finding discovery (M2), and tool usage appropriateness (M3), assessed through an adversarial LLM-as-Judge that inverts the burden of proof -- requiring concrete forensic evidence to credit investigations. Evaluating our SIR agent on the benchmark demonstrates 97.1% true positive (TP) detection, 73.4% false positive (FP) rejection, and 5.67 novel key findings per case, establishing a baseline against which future investigation agents can be measured.