Ranking Algorithms by Performance
This work addresses algorithm selection limitations for researchers and practitioners in machine learning, offering an incremental improvement by extending single-algorithm prediction to rankings.
The paper tackles the problem of algorithm selection by predicting the ranking of portfolio algorithms on a given problem, enabling better resource allocation and parallelization. It shows that considering relationships between algorithms improves ranking predictions, with naive approaches already delivering good quality results.
A common way of doing algorithm selection is to train a machine learning model and predict the best algorithm from a portfolio to solve a particular problem. While this method has been highly successful, choosing only a single algorithm has inherent limitations -- if the choice was bad, no remedial action can be taken and parallelism cannot be exploited, to name but a few problems. In this paper, we investigate how to predict the ranking of the portfolio algorithms on a particular problem. This information can be used to choose the single best algorithm, but also to allocate resources to the algorithms according to their rank. We evaluate a range of approaches to predict the ranking of a set of algorithms on a problem. We furthermore introduce a framework for categorizing ranking predictions that allows to judge the expressiveness of the predictive output. Our experimental evaluation demonstrates on a range of data sets from the literature that it is beneficial to consider the relationship between algorithms when predicting rankings. We furthermore show that relatively naive approaches deliver rankings of good quality already.