Comparing Mutation Coverage Against Branch Coverage in an Industrial Setting
This addresses the problem of improving test suite quality for software development teams in industrial settings, but it is incremental as it focuses on adoption barriers rather than a new solution.
The paper investigated why mutation coverage is not widely adopted in continuous integration settings compared to branch coverage, finding that it reveals additional weaknesses in test suites with acceptable performance overhead in a case study involving four open-source and one industrial system.
The state-of-the-practice in software development is driven by constant change fueled by continuous integration servers. Such constant change demands for frequent and fully automated tests capable to detect faults immediately upon project build. As the fault detection capability of the test suite becomes so important, modern software development teams continuously monitor the quality of the test suite as well. However, it appears that the state-of-the-practice is reluctant to adopt strong coverage metrics (namely mutation coverage), instead relying on weaker kinds of coverage (namely branch coverage). In this paper, we investigate three reasons that prohibit the adoption of mutation coverage in a continuous integration setting: (1) the difficulty of its integration into the build system, (2) the perception that branch coverage is "good enough", and (3) the performance overhead during the build. Our investigation is based on a case study involving four open source systems and one industrial system. We demonstrate that mutation coverage reveals additional weaknesses in the test suite compared to branch coverage and that it is able to do so with an acceptable performance overhead during project build.