Differentially Testing Soundness and Precision of Program Analyzers
This addresses the problem for designers and users of program analyzers by providing a way to compare effectiveness, though it is incremental as it builds on existing testing methods.
The paper tackled the lack of systematic comparison for program analyzers by developing an automated technique to differentially test their soundness and precision, detecting issues in most of six state-of-the-art analyzers on tens of thousands of benchmarks.
In the last decades, numerous program analyzers have been developed both by academia and industry. Despite their abundance however, there is currently no systematic way of comparing the effectiveness of different analyzers on arbitrary code. In this paper, we present the first automated technique for differentially testing soundness and precision of program analyzers. We used our technique to compare six mature, state-of-the art analyzers on tens of thousands of automatically generated benchmarks. Our technique detected soundness and precision issues in most analyzers, and we evaluated the implications of these issues to both designers and users of program analyzers.