CASTLE: Benchmarking Dataset for Static Code Analyzers and LLMs towards CWE Detection
This work provides a standardized benchmark for comparing security flaw detection tools, aiding developers and researchers in evaluating and improving vulnerability identification methods, though it is incremental as it builds on existing evaluation practices.
The paper introduces CASTLE, a benchmarking framework for evaluating vulnerability detection methods like static analyzers, LLMs, and formal verification tools on a dataset of 250 micro-benchmark programs covering 25 CWEs, revealing that LLMs perform well on small code snippets but decline with larger code, while static analyzers have high false positives and formal verification tools like ESBMC minimize false positives but struggle with certain vulnerabilities.
Identifying vulnerabilities in source code is crucial, especially in critical software components. Existing methods such as static analysis, dynamic analysis, formal verification, and recently Large Language Models are widely used to detect security flaws. This paper introduces CASTLE (CWE Automated Security Testing and Low-Level Evaluation), a benchmarking framework for evaluating the vulnerability detection capabilities of different methods. We assess 13 static analysis tools, 10 LLMs, and 2 formal verification tools using a hand-crafted dataset of 250 micro-benchmark programs covering 25 common CWEs. We propose the CASTLE Score, a novel evaluation metric to ensure fair comparison. Our results reveal key differences: ESBMC (a formal verification tool) minimizes false positives but struggles with vulnerabilities beyond model checking, such as weak cryptography or SQL injection. Static analyzers suffer from high false positives, increasing manual validation efforts for developers. LLMs perform exceptionally well in the CASTLE dataset when identifying vulnerabilities in small code snippets. However, their accuracy declines, and hallucinations increase as the code size grows. These results suggest that LLMs could play a pivotal role in future security solutions, particularly within code completion frameworks, where they can provide real-time guidance to prevent vulnerabilities. The dataset is accessible at https://github.com/CASTLE-Benchmark.