An empirical study of public data quality problems in cross project defect prediction
This addresses data reliability for researchers in software engineering using cross-project defect prediction, but it is incremental as it builds on prior work on data quality.
The study investigated data quality issues, specifically identical and inconsistent cases, in the widely used Jureczko datasets for cross-project defect prediction, finding that cleaning these datasets led to very different performance results in terms of AUC and F-Measure.
Background: Two public defect data, including Jureczko and NASA datasets, have been widely used in cross project defect prediction (CPDP). The quality of defect data have been reported as an important factor influencing the defect prediction performance and Shepperd et al. have researched the data quality problems in NASA datasets. However, up to now, there is no research focusing on the quality problems of Jureczko datasets which are most widely used in CPDP. Aims: In this paper, we intend to investigate the problems of identical and inconsistent cases in Jureczko datasets and validate whether removing these problematic cases will make a difference to defect prediction performance in CPDP. Method: The problems of identical and inconsistent cases are reported from two aspects, respectively in each individual dataset and in a pair of datasets from different releases of a software project. Then a cleaned version of Jureczko datasets is provided by removing duplicate and inconsistent cases. Finally three training data selection methods are employed to compare the defect prediction performance of cleaned datasets with that of original datasets. Results: The experimental results in terms of AUC and F-Measure show that most datasets obtain very different defect prediction performance. Conclusions: It is very necessary to study the data quality problems in CPDP and the cleaned Jureczko datasets may provide more reliable defect prediction performance in CPDP.