Improving LLM Agents with Reinforcement Learning on Cryptographic CTF Challenges
This work addresses the challenge of building intelligent and adaptable LLM agents for complex cybersecurity tasks, representing an incremental advancement through fine-tuning and dataset creation.
The paper tackled the problem of improving LLM-based agents for security-sensitive domains by using reinforcement learning on a procedurally generated cryptographic CTF dataset, resulting in significant improvements in Pass@8 on unseen challenges and generalization to external benchmarks like picoCTF and AICrypto MCQ.
We present 'Random-Crypto', a procedurally generated cryptographic Capture The Flag (CTF) dataset designed to unlock the potential of Reinforcement Learning (RL) for LLM-based agents in security-sensitive domains. Cryptographic reasoning offers an ideal RL testbed: it combines precise validation, structured multi-step inference, and reliance on reliable computational tool use. Leveraging these properties, we fine-tune a Python tool-augmented Llama-3.1-8B via Group Relative Policy Optimization (GRPO) in a secure execution environment. The resulting agent achieves a significant improvement in Pass@8 on previously unseen challenges. Moreover, the improvements generalize to two external benchmarks: 'picoCTF', spanning both crypto and non-crypto tasks, and 'AICrypto MCQ', a multiple-choice benchmark of 135 cryptography questions. Ablation studies attribute the gains to enhanced tool usage and procedural reasoning. These findings position 'Random-Crypto' as a rich training ground for building intelligent, adaptable LLM agents capable of handling complex cybersecurity tasks.