Assessing and Improving the Mutation Testing Practice of PIT
This work addresses a gap in mutation testing practice for software engineers, highlighting limitations in widely used tools, but it is incremental as it builds on existing literature.
The study evaluated the effectiveness of mutants generated by the PIT mutation testing tool compared to comprehensive mutants from literature, finding that comprehensive mutants are harder to kill and capture faults missed by PIT in 11% to 62% of Java classes across projects.
Mutation testing is used extensively to support the experimentation of software engineering studies. Its application to real-world projects is possible thanks to modern tools that automate the whole mutation analysis process. However, popular mutation testing tools use a restrictive set of mutants which do not conform to the community standards as supported by the mutation testing literature. This can be problematic since the effectiveness of mutation depends on its mutants. We therefore examine how effective are the mutants of a popular mutation testing tool, named PIT, compared to comprehensive ones, as drawn from the literature and personal experience. We show that comprehensive mutants are harder to kill and encode faults not captured by the mutants of PIT for a range of 11% to 62% of the Java classes of the considered projects.