Statistical Multicriteria Benchmarking via the GSD-Front
This addresses the need for robust benchmarking in machine learning, particularly for researchers and practitioners comparing classifiers, though it is incremental as it builds on existing concepts like stochastic dominance.
The paper tackles the problem of reliably comparing classifiers by proposing a method that allows for multiple quality metrics, accounts for statistical uncertainty, and verifies robustness under assumption deviations, resulting in a statistical test for whether a new classifier lies in the GSD-front of state-of-the-art ones.
Given the vast number of classifiers that have been (and continue to be) proposed, reliable methods for comparing them are becoming increasingly important. The desire for reliability is broken down into three main aspects: (1) Comparisons should allow for different quality metrics simultaneously. (2) Comparisons should take into account the statistical uncertainty induced by the choice of benchmark suite. (3) The robustness of the comparisons under small deviations in the underlying assumptions should be verifiable. To address (1), we propose to compare classifiers using a generalized stochastic dominance ordering (GSD) and present the GSD-front as an information-efficient alternative to the classical Pareto-front. For (2), we propose a consistent statistical estimator for the GSD-front and construct a statistical test for whether a (potentially new) classifier lies in the GSD-front of a set of state-of-the-art classifiers. For (3), we relax our proposed test using techniques from robust statistics and imprecise probabilities. We illustrate our concepts on the benchmark suite PMLB and on the platform OpenML.