Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks
For cybersecurity researchers and practitioners, this provides the most comprehensive cross-model evaluation of LLM agents on offensive tasks, revealing that environment tooling and model selection are the strongest performance drivers.
This paper benchmarks 10 frontier LLMs on 200 offensive cybersecurity tasks, finding that Claude 4.5 Opus achieves the highest solve rate (59%) and Gemini 3 Flash offers the best cost-efficiency ($0.05 per solve). The Kali Linux environment improves performance by +9.5 percentage points over Ubuntu, while prompt engineering often degrades performance in well-equipped environments.
We present, to our knowledge, the most comprehensive cross-model evaluation of LLM agents on offensive cybersecurity tasks, benchmarking 10 frontier models from 7 providers on all 200 challenges of the NYU CTF Bench. Building on the D-CIPHER multi-agent framework, we extend it with multi-provider backend support, a custom Kali Linux environment with over 100 pre-installed penetration testing tools, and runtime tool-discovery agents. Through a controlled factorial study, we find that the Kali Linux environment yields a +9.5 percentage-point improvement over Ubuntu, while auto-prompting and category-specific tips often degrade performance in well-equipped environments. Among models, Claude 4.5 Opus achieves the highest solve rate (59%), followed by Gemini 3 Pro (52%), with Gemini 3 Flash offering the best cost-efficiency at $0.05 per solve. Asymmetric planner/executor model assignments provide no meaningful benefit while coherent same-model configurations consistently outperform mixed-tier pairings. Our results indicate that environment tooling and model selection emerge as the strongest drivers of performance, whereas prompt engineering interventions show diminishing or negative returns in well-equipped environments. Reported performance reflects both model reasoning ability and compatibility with agent tooling and API integration.