Towards Optimal Agentic Architectures for Offensive Security Tasks
This work provides empirical guidance for practitioners designing agentic security systems by systematically comparing topology choices, showing that more agents do not always yield better cost-quality trade-offs.
The paper investigates how different agentic architectures (topologies) affect the cost and effectiveness of LLM-based security auditing systems. On a benchmark of 20 interactive targets, the best validated detection rate (64.2%) was achieved by a multi-agent independent topology, while a single-agent system was the most efficient at $0.058 per validated finding. The main finding is a non-monotonic cost-quality frontier where broader coordination improves coverage but does not dominate when considering latency, cost, and validation difficulty.
Agentic security systems increasingly audit live targets with tool-using LLMs, but prior systems fix a single coordination topology, leaving unclear when additional agents help and when they only add cost. We treat topology choice as an empirical systems question. We introduce a controlled benchmark of 20 interactive targets (10 web/API and 10 binary), each exposing one endpoint-reachable ground-truth vulnerability, evaluated in whitebox and blackbox modes. The core study executes 600 runs over five architecture families, three model families, and both access modes, with a separate 60-run long-context pilot reported only in the appendix. On the completed core benchmark, detection-any reaches 58.0% and validated detection reaches 49.8%. MAS-Indep attains the highest validated detection rate (64.2%), while SAS is the strongest efficiency baseline at $0.058 per validated finding. Whitebox materially outperforms blackbox (67.0% vs. 32.7% validated detection), and web materially outperforms binary (74.3% vs. 25.3%). Bootstrap confidence intervals and paired target-level deltas show that the dominant effects are observability and domain, while some leading whitebox topologies remain statistically close. The main result is a non-monotonic cost-quality frontier: broader coordination can improve coverage, but it does not dominate once latency, token cost, and exploit-validation difficulty are taken into account.