Open-source Defect Injection Benchmark Testbed for the Evaluation of Testing
This provides a tool for software testing researchers and practitioners to evaluate testing effectiveness, though it is incremental as it builds on existing defect injection methods.
The authors tackled the problem of evaluating testing techniques by developing an open-source benchmark testbed that injects artificial defects into source code using predefined building blocks, complementing mutation testing to simulate complex defects, and it includes automated test cases covering most functions and UI elements.
A natural method to evaluate the effectiveness of a testing technique is to measure the defect detection rate when applying the created test cases. Here, real or artificial software defects can be injected into the source code of software. For a more extensive evaluation, the injection of artificial defects is usually needed and can be performed via mutation testing using code mutation operators. However, to simulate complex defects arising from a misunderstanding of design specifications, mutation testing might reach its limit in some cases. In this paper, we present an open-source benchmark testbed application that employs a complement method of artificial defect injection. The application is compiled after artificial defects are injected into its source code from predefined building blocks. The majority of the functions and user interface elements are covered by creating front-end-based automated test cases that can be used in experiments.