Data-Centric Benchmarking of Exploit Generation in LLMs: Understanding the Impact of Fine-Tuning
For cybersecurity practitioners, this work demonstrates that data quality and fine-tuning can significantly boost LLM performance in exploit generation, challenging the primacy of model scale.
The authors constructed a high-quality dataset and evaluation framework for CVE-conditioned exploit generation, then benchmarked 17 LLMs. Fine-tuning an 8B open-weight model on curated data improved exploit quality by over 42.5%, rivaling proprietary models with test-time rejection.
We study the task of CVE-conditioned exploit generation, where a model drafts proof-of-concept (PoC) exploits given software vulnerability context. We adopt a data-centric approach, constructing a high-quality dataset via multi-stage preprocessing and introducing a scalable evaluation framework with LLM-as-judge and fine-grained rubrics. Under this unified setup, we benchmark 17 large language models across 8 evaluation criteria, providing systematic insights into their zero-shot capabilities. We further show that a compact 8B open-weight model, when fine-tuned on curated data, achieves over 42.5% improvement in exploit quality and rivals some proprietary models when combined with simple test-time rejection strategies. Our results highlight the importance of data quality, structured supervision, and evaluation design for reliable exploit generation, suggesting that these factors can be as critical as model scale in adapting LLMs to cybersecurity tasks.