Classifier Selection with Permutation Tests
This work addresses the challenge of classifier selection for machine learning practitioners, offering an incremental improvement over existing methods by incorporating robust statistical tests.
The paper tackles the problem of selecting the best classifier for a new dataset by developing a content-based recommender system that uses permutation tests to assess classifier performance, showing that this approach improves recommendation quality based on experiments with 8 classifiers and 65 datasets.
This work presents a content-based recommender system for machine learning classifier algorithms. Given a new data set, a recommendation of what classifier is likely to perform best is made based on classifier performance over similar known data sets. This similarity is measured according to a data set characterization that includes several state-of-the-art metrics taking into account physical structure, statis- tics, and information theory. A novelty with respect to prior work is the use of a robust approach based on permutation tests to directly assess whether a given learning algorithm is able to exploit the attributes in a data set to predict class labels, and compare it to the more commonly used F-score metric for evalu- ating classifier performance. To evaluate our approach, we have conducted an extensive experimentation including 8 of the main machine learning classification methods with varying configurations and 65 bi- nary data sets, leading to over 2331 experiments. Our results show that using the information from the permutation test clearly improves the quality of the recommendations.