Causal Testing: Finding Defects' Root Causes
This addresses the challenge for software developers in debugging by providing a novel tool for root-cause analysis, though it is incremental as it builds on existing causality theory.
The paper tackles the problem of identifying root causes of software defects by introducing Causal Testing, a method based on counterfactual causality, and shows it can be applied to 71% of real-world defects in the Defects4J benchmark, helping developers identify root causes 77% of the time and improving identification accuracy from 80% to 86% in a controlled experiment.
Understanding the root cause of a defect is critical to isolating and repairing buggy behavior. We present Causal Testing, a new method of root-cause analysis that relies on the theory of counterfactual causality to identify a set of executions that likely hold key causal information necessary to understand and repair buggy behavior. Using the Defects4J benchmark, we find that Causal Testing could be applied to 71% of real-world defects, and for 77% of those, it can help developers identify the root cause of the defect. A controlled experiment with 37 developers shows that Causal Testing improves participants' ability to identify the cause of the defect from 80% of the time with standard testing tools to 86% of the time with Causal Testing. The participants report that Causal Testing provides useful information they cannot get using tools such as JUnit. Holmes, our prototype, open-source Eclipse plugin implementation of Causal Testing, is available at http://holmes.cs.umass.edu/.