Supporting Artifact Evaluation with LLMs: A Study with Published Security Research Papers
This work addresses the scalability issue in artifact evaluation for cybersecurity conferences, particularly in IoT and CPS domains, by reducing reviewer effort and potentially improving artifact quality, though it is incremental as it builds on existing AE processes.
The study tackled the problem of time-consuming manual artifact evaluation in cybersecurity research by developing an LLM-based toolkit that automates reproducibility rating, environment setup, and pitfall assessment, achieving over 72% accuracy in reproducibility rating and autonomously setting up environments for 28% of runnable artifacts.
Artifact Evaluation (AE) is essential for ensuring the transparency and reliability of research, closing the gap between exploratory work and real-world deployment is particularly important in cybersecurity, particularly in IoT and CPSs, where large-scale, heterogeneous, and privacy-sensitive data meet safety-critical actuation. Yet, manual reproducibility checks are time-consuming and do not scale with growing submission volumes. In this work, we demonstrate that Large Language Models (LLMs) can provide powerful support for AE tasks: (i) text-based reproducibility rating, (ii) autonomous sandboxed execution environment preparation, and (iii) assessment of methodological pitfalls. Our reproducibility-assessment toolkit yields an accuracy of over 72% and autonomously sets up execution environments for 28% of runnable cybersecurity artifacts. Our automated pitfall assessment detects seven prevalent pitfalls with high accuracy ($F_1$ > 92%). Hence, the toolkit significantly reduces reviewer effort and, when integrated into established AE processes, could incentivize authors to submit higher-quality and more reproducible artifacts. IoT, CPS, and cybersecurity conferences and workshops may integrate the toolkit into their peer-review processes to support reviewers' decisions on awarding artifact badges, improving the overall sustainability of the process.