An Approach to Multiple Comparison Benchmark Evaluations that is Stable Under Manipulation of the Comparate Set
This addresses a methodological issue for researchers and practitioners in machine learning and computer science who rely on benchmark comparisons, though it is incremental rather than foundational.
The paper tackles the problem of benchmark evaluation methods being vulnerable to manipulation, and proposes a new presentation approach called Multiple Comparison Matrix (MCM) that prevents such manipulation by prioritizing pairwise comparisons.
The measurement of progress using benchmarks evaluations is ubiquitous in computer science and machine learning. However, common approaches to analyzing and presenting the results of benchmark comparisons of multiple algorithms over multiple datasets, such as the critical difference diagram introduced by Demšar (2006), have important shortcomings and, we show, are open to both inadvertent and intentional manipulation. To address these issues, we propose a new approach to presenting the results of benchmark comparisons, the Multiple Comparison Matrix (MCM), that prioritizes pairwise comparisons and precludes the means of manipulating experimental results in existing approaches. MCM can be used to show the results of an all-pairs comparison, or to show the results of a comparison between one or more selected algorithms and the state of the art. MCM is implemented in Python and is publicly available.