Builder, Defender, Breaker: The Case Against Removing the Human from the AI-Driven Security Lifecycle
For security practitioners and AI system designers, this paper challenges the trend toward full autonomy in security, arguing it undermines verification and accountability.
The paper argues that fully automating the AI-driven security lifecycle (building, defending, and testing code) with the same generative model collapses the independence needed for verification, introduces shared blind spots, and removes human accountability. It advocates for keeping humans permanently in the loop based on evidence from code generation, adversarial ML, and all-machine hacking tournaments.
Artificial intelligence has spread across the whole of the security lifecycle. The same family of models now writes application code, hardens it, and probes it for weaknesses, so that a single generative substrate increasingly performs all three roles at once. Enthusiasm for this convergence tends to treat full autonomy as the natural end point of partial assistance. This article argues that it is not. When the system that builds an artifact is drawn from the same distribution as the systems that defend and test it, the three roles inherit a common set of blind spots, and the independence that makes verification meaningful is quietly lost. Removing the human does more than raise the automation level: it collapses the external oracle against which machine output is judged, outruns the point at which a person could intervene, hands adversaries a predictable and poisonable target, and dissolves the locus of accountability when something fails. Drawing on evidence from autonomous code generation, adversarial machine learning, software fault tolerance, and the first all-machine hacking tournaments, we argue that the human belongs in the loop not as a temporary scaffold but as a permanent structural requirement, and set out what a defensible division of labour between people and machines should preserve.