Multi-objective Analysis of MAP-Elites Performance
This provides a method for multi-objective performance analysis in optimization, but it is incremental as it builds on existing techniques like Pareto dominance and ordinal analysis.
The paper tackles the problem of analyzing algorithm performance in complex optimization tasks with multiple measures by introducing a method combining ordinal effect sizes and Pareto dominance, which generalizes across tasks with differently scaled measurements. In a case study on MAP-Elites for evolving robot controllers, it shows that average mutation magnitude has a bigger effect on outcomes than precise distributions.
In certain complex optimization tasks, it becomes necessary to use multiple measures to characterize the performance of different algorithms. This paper presents a method that combines ordinal effect sizes with Pareto dominance to analyze such cases. Since the method is ordinal, it can also generalize across different optimization tasks even when the performance measurements are differently scaled. Through a case study, we show that this method can discover and quantify relations that would be difficult to deduce using a conventional measure-by-measure analysis. This case study applies the method to the evolution of robot controller repertoires using the MAP-Elites algorithm. Here, we analyze the search performance across a large set of parametrizations; varying mutation size and operator type, as well as map resolution, across four different robot morphologies. We show that the average magnitude of mutations has a bigger effect on outcomes than their precise distributions.