A systematic literature review on the code smells datasets and validation mechanisms
This work identifies dataset limitations that affect tool evaluation in software engineering, but it is incremental as it reviews existing data without proposing new solutions.
The paper surveyed 45 datasets for code smell detection and found that their adequacy depends on properties like size and severity, with most supporting only a few smells and suffering from imbalances and language restrictions.
The accuracy reported for code smell-detecting tools varies depending on the dataset used to evaluate the tools. Our survey of 45 existing datasets reveals that the adequacy of a dataset for detecting smells highly depends on relevant properties such as the size, severity level, project types, number of each type of smell, number of smells, and the ratio of smelly to non-smelly samples in the dataset. Most existing datasets support God Class, Long Method, and Feature Envy while six smells in Fowler and Beck's catalog are not supported by any datasets. We conclude that existing datasets suffer from imbalanced samples, lack of supporting severity level, and restriction to Java language.