Adaptive Local Bayesian Optimization Over Multiple Discrete Variables
This work provides an incremental improvement in automated hyperparameter optimization for machine learning practitioners, particularly for black-box functions.
This paper addresses the challenge of automated hyperparameter optimization for black-box functions, proposing an approach that combines local Bayesian search, a combinatorial Gaussian process kernel, and a hybrid Bayesian/multi-armed bandit strategy. The method achieved up to a +20.39% improvement over baseline algorithms in the NeurIPS 2020 Black-Box Optimization challenge.
In the machine learning algorithms, the choice of the hyperparameter is often an art more than a science, requiring labor-intensive search with expert experience. Therefore, automation on hyperparameter optimization to exclude human intervention is a great appeal, especially for the black-box functions. Recently, there have been increasing demands of solving such concealed tasks for better generalization, though the task-dependent issue is not easy to solve. The Black-Box Optimization challenge (NeurIPS 2020) required competitors to build a robust black-box optimizer across different domains of standard machine learning problems. This paper describes the approach of team KAIST OSI in a step-wise manner, which outperforms the baseline algorithms by up to +20.39%. We first strengthen the local Bayesian search under the concept of region reliability. Then, we design a combinatorial kernel for a Gaussian process kernel. In a similar vein, we combine the methodology of Bayesian and multi-armed bandit,(MAB) approach to select the values with the consideration of the variable types; the real and integer variables are with Bayesian, while the boolean and categorical variables are with MAB. Empirical evaluations demonstrate that our method outperforms the existing methods across different tasks.