The Cost and Benefits of Static Analysis During Development
It provides real-world benchmarks and cost-effectiveness analysis for software developers and managers, addressing a gap in quantitative data for engineering trade-offs, though it is incremental in applying existing methods to new data.
This study tackled the problem of quantifying the cost and benefits of static analysis in software development by analyzing data from 35 industrial projects, finding that using static analysis tools increased defect removal rates and reduced both final defect density and total development effort.
Without quantitative data, deciding whether and how to use static analysis in a development workflow is a matter of expert opinion and guesswork rather than an engineering trade-off. Moreover, relevant data collected under real-world conditions is scarce. Important but unknown quantitative parameters include, but are not limited to, the effort to apply the techniques, the effectiveness of removing defects, where in the workflow the analysis should be applied, and how static analysis interacts with other quality techniques. This study examined the detailed development process data 35 industrial development projects that included static analysis and that were also instrumented with the Team Software Process. We collected data project plans, logs of effort, defect, and size and post mortem reports and analyzed performance of their development activities to populate a parameterized performance model. We compared effort and defect levels with and without static analysis using a planning model that includes feedback for defect removal effectiveness and fix effort. We found evidence that using each tool developers found and removed defects at a higher rate than alternative removal techniques. Moreover, the early and inexpensive removal reduced not only final defect density but also total development effort. The contributions of this paper include real-world benchmarks of process data from projects using static analysis tools, a demonstration of a cost-effectiveness analysis using this data, and a recommendation these tools were consistently cost effective operationally.