Experimental Identification of Hard Data Sets for Classification and Feature Selection Methods with Insights on Method Selection
This work addresses the problem of selecting appropriate classification and feature selection methods for researchers and practitioners by benchmarking on hard data sets, though it is incremental as it builds on existing methods and data.
The paper identifies 15 out of 129 UCI data sets as hard for classification and feature selection methods, where no method combination achieved AUC or F-Measure over 0.8, and provides new rankings of methods on hard vs. easy data sets, with Random Forest consistently performing best.
The paper reports an experimentally identified list of benchmark data sets that are hard for representative classification and feature selection methods. This was done after systematically evaluating a total of 48 combinations of methods, involving eight state-of-the-art classification algorithms and six commonly used feature selection methods, on 129 data sets from the UCI repository (some data sets with known high classification accuracy were excluded). In this paper, a data set for classification is called hard if none of the 48 combinations can achieve an AUC over 0.8 and none of them can achieve an F-Measure value over 0.8; it is called easy otherwise. A total of 15 out of the 129 data sets were found to be hard in that sense. This paper also compares the performance of different methods, and it produces rankings of classification methods, separately on the hard data sets and on the easy data sets. This paper is the first to rank methods separately for hard data sets and for easy data sets. It turns out that the classifier rankings resulting from our experiments are somehow different from those in the literature and hence they offer new insights on method selection. It should be noted that the Random Forest method remains to be the best in all groups of experiments.