Tuning as a Means of Assessing the Benefits of New Ideas in Interplay with Existing Algorithmic Modules
This work addresses a methodological gap for researchers in optimization algorithms, though it is incremental as it builds on existing frameworks.
The paper tackles the challenge of assessing new algorithmic components in optimization by introducing a hyperparameter tuning procedure to evaluate their interplay with existing modules, implemented in the Modular CMA-ES framework to analyze step-size adaptation methods and identify conditions for their effectiveness.
Introducing new algorithmic ideas is a key part of the continuous improvement of existing optimization algorithms. However, when introducing a new component into an existing algorithm, assessing its potential benefits is a challenging task. Often, the component is added to a default implementation of the underlying algorithm and compared against a limited set of other variants. This assessment ignores any potential interplay with other algorithmic ideas that share the same base algorithm, which is critical in understanding the exact contributions being made. We introduce a more extensive procedure, which uses hyperparameter tuning as a means of assessing the benefits of new algorithmic components. This allows for a more robust analysis by not only focusing on the impact on performance, but also by investigating how this performance is achieved. We implement our suggestion in the context of the Modular CMA-ES framework, which was redesigned and extended to include some new modules and several new options for existing modules, mostly focused on the step-size adaptation method. Our analysis highlights the differences between these new modules, and identifies the situations in which they have the largest contribution.