What Makes Software Bugs Escape Testing? Evidence from a Large-Scale Empirical Study
For software engineers and testers, this study identifies key factors (evolutionary and process dynamics) that cause bugs to escape testing, guiding targeted testing efforts.
Analyzed over 14k defects from open-source projects to characterize pre- and post-release bugs, finding that post-release defects are concentrated in older, frequently modified, high-churn components and require longer, more complex fixes.
Understanding how software defects manifest and evolve in production environments is critical for improving reliability. While previous research has largely focused on pre-release defects, the nature of residual faults, i.e., those escaping testing and surfacing post-release, remains poorly understood. This paper presents a large-scale characterization of pre- and post-release defects across C/C++ and Java systems, encompassing over 14k defects mined from open-source projects. We employ a broad suite of software metrics to capture diverse code attributes such as complexity, size, structure, and development history. Results show that post-release defects are concentrated in older, frequently modified, and high-churn components, typically requiring longer and more complex fixes than pre-release ones. These findings highlight that residual defects arise more from evolutionary and process dynamics than code structure alone, suggesting that reliability engineering should prioritize targeted testing in mature and complex code regions.