Hacking CTFs with Plain Agents
This demonstrates that current LLMs have surpassed high school-level offensive cybersecurity capabilities, potentially impacting security testing and education.
The researchers tackled the problem of automated hacking by achieving 95% performance on the InterCode-CTF benchmark using plain LLM agents with prompting, tool use, and multiple attempts, beating prior work by significant margins (29% and 72%).
We saturate a high-school-level hacking benchmark with plain LLM agent design. Concretely, we obtain 95% performance on InterCode-CTF, a popular offensive security benchmark, using prompting, tool use, and multiple attempts. This beats prior work by Phuong et al. 2024 (29%) and Abramovich et al. 2024 (72%). Our results suggest that current LLMs have surpassed the high school level in offensive cybersecurity. Their hacking capabilities remain underelicited: our ReAct&Plan prompting strategy solves many challenges in 1-2 turns without complex engineering or advanced harnessing.