On the Distribution of "Simple Stupid Bugs" in Unit Test Files: An Exploratory Study
This work addresses the problem of test code quality for software developers by identifying bug patterns, though it is incremental as it builds on existing research on SStuBs and test smells.
The study compared the frequency of 'simple stupid bugs' (SStuBs) in test versus production files in Java Maven projects, finding that SStuBs occur more often in non-test files, with most fixes in test files related to assertions and often co-occurring with test smells.
A key aspect of ensuring the quality of a software system is the practice of unit testing. Through unit tests, developers verify the correctness of production source code, thereby verifying the system's intended behavior under test. However, unit test code is subject to issues, ranging from bugs in the code to poor test case design (i.e., test smells). In this study, we compare and contrast the occurrences of a type of single-statement-bug-fix known as "simple stupid bugs" (SStuBs) in test and non-test (i.e., production) files in popular open-source Java Maven projects. Our results show that SStuBs occur more frequently in non-test files than in test files, with most fix-related code associated with assertion statements in test files. Further, most test files exhibiting SStuBs also exhibit test smells. We envision our findings enabling tool vendors to better support developers in improving the maintenance of test suites.