Keno Hassler, Philipp Görz, Stephan Lipp
Over 70% of security vulnerabilities in critical software systems today result from memory safety violations. To address this challenge, fuzzing and static analysis are widely used automated methods to discover such vulnerabilities. Fuzzing generates random program inputs to identify faults at runtime, while static analysis reasons about the code to detect potential vulnerabilities. Although these techniques share a common goal, they take fundamentally different approaches and have evolved largely independently. In this paper, we present an empirical analysis of five static analyzers and 13 fuzzers, applied to over 100 known security vulnerabilities in C/C++ programs. We measure the detection rate for each tool and vulnerability to evaluate how the approaches differ and complement each other. We find that fuzzers discover a very similar set of bugs, while static analyzers report more diverse sets, and identify clear leaders for each group. Comparing the union of all fuzzers with that of all static analyzers, we observe they are nearly disjoint. In a second step, we manually validate the report-to-bug mapping we developed for the evaluation and discuss more qualitative aspects of limitations, usability, and integration into the development process. We examine how widely these bug finding tools are used in critical open-source projects. We advise developers on choosing tools to harden their software and identify barriers to adoption as well as future research opportunities.