Assessing Exception Handling Testing Practices in Open-Source Libraries
This addresses software reliability issues for developers by evaluating testing effectiveness in exception handling, though it is incremental as it builds on prior studies of testing practices.
The study assessed exception handling testing practices in 27 open-source Java libraries, finding that catch blocks and throw instructions have significantly lower code coverage than overall code, but test suites still detected 68% of 12,331 artificially injected faults.
Modern programming languages (e.g., Java and C#) provide features to separate error-handling code from regular code, seeking to enhance software comprehensibility and maintainability. Nevertheless, the way exception handling (EH) code is structured in such languages may lead to multiple, different, and complex control flows, which may affect the software testability. Previous studies have reported that EH code is typically neglected, not well tested, and its misuse can lead to reliability degradation and catastrophic failures. However, little is known about the relationship between testing practices and EH testing effectiveness. In this exploratory study, we (i) measured the adequacy degree of EH testing concerning code coverage (instruction, branch, and method) criteria; and (ii) evaluated the effectiveness of the EH testing by measuring its capability to detect artificially injected faults (i.e., mutants) using 7 EH mutation operators. Our study was performed using test suites of 27 long-lived Java libraries from open-source ecosystems. Our results show that instructions and branches within $\mathtt{catch}$ blocks and $\mathtt{throw}$ instructions are less covered, with statistical significance than the overall instructions and branches. Nevertheless, most of the studied libraries presented test suites capable of detecting more than 70% of the injected faults. From a total of 12,331 mutants created in this study, the test suites were able to detect 68% of them.