COCO: Performance Assessment
This provides a standardized benchmarking method for researchers in numerical optimization, though it is incremental as it builds on existing COCO platform concepts.
The paper tackles the problem of benchmarking numerical optimization algorithms in black-box scenarios by proposing a performance assessment method based on runtime measured in objective function evaluations to reach quality targets. The result is a framework implemented within the COCO platform that uses simulated restarts and empirical distribution functions for result aggregation.
We present an any-time performance assessment for benchmarking numerical optimization algorithms in a black-box scenario, applied within the COCO benchmarking platform. The performance assessment is based on runtimes measured in number of objective function evaluations to reach one or several quality indicator target values. We argue that runtime is the only available measure with a generic, meaningful, and quantitative interpretation. We discuss the choice of the target values, runlength-based targets, and the aggregation of results by using simulated restarts, averages, and empirical distribution functions.