Who is the Winning Algorithm? Rank Aggregation for Comparative Studies
This work addresses the need for more accurate algorithm selection in machine learning benchmarking, though it is incremental as it builds on existing rank aggregation methods.
The paper tackles the problem of identifying the best-performing machine learning algorithm from comparative studies by estimating win probabilities using complete rankings, and it shows significant improvements over existing methods in synthetic and real-world examples.
Consider a collection of m competing machine learning algorithms. Given their performance on a benchmark of datasets, we would like to identify the best performing algorithm. Specifically, which algorithm is most likely to ``win'' (rank highest) on a future, unseen dataset. The standard maximum likelihood approach suggests counting the number of wins per each algorithm. In this work, we argue that there is much more information in the complete rankings. That is, the number of times that each algorithm finished second, third and so forth. Yet, it is not entirely clear how to effectively utilize this information for our purpose. In this work we introduce a novel conceptual framework for estimating the win probability for each of the m algorithms, given their complete rankings over a benchmark of datasets. Our proposed framework significantly improves upon currently known methods in synthetic and real-world examples.