Evaluation of Static Vulnerability Detection Tools with Java Cryptographic API Benchmarks
This work addresses the need for reliable benchmarks to compare security tools for developers and researchers, but it is incremental as it builds on existing evaluation methods without introducing new detection techniques.
The researchers tackled the problem of evaluating static vulnerability detection tools for Java cryptographic API misuses by developing two benchmarks, CryptoAPI-Bench with 181 test cases and ApacheCryptoAPI-Bench with 121 cases from Apache projects, and found that tools like SpotBugs, CryptoGuard, CrySL, and Coverity showed varying performance in accuracy and scalability.
Several studies showed that misuses of cryptographic APIs are common in real-world code (e.g., Apache projects and Android apps). There exist several open-sourced and commercial security tools that automatically screen Java programs to detect misuses. To compare their accuracy and security guarantees, we develop two comprehensive benchmarks named CryptoAPI-Bench and ApacheCryptoAPI-Bench. CryptoAPI-Bench consists of 181 unit test cases that cover basic cases, as well as complex cases, including interprocedural, field sensitive, multiple class test cases, and path sensitive data flow of misuse cases. The benchmark also includes correct cases for testing false-positive rates. The ApacheCryptoAPI-Bench consists of 121 cryptographic cases from 10 Apache projects. We evaluate four tools, namely, SpotBugs, CryptoGuard, CrySL, and Coverity using both benchmarks. We present their performance and comparative analysis. The ApacheCryptoAPI-Bench also examines the scalability of the tools. Our benchmarks are useful for advancing state-of-the-art solutions in the space of misuse detection.