Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spaces
This work addresses a specific challenge in hyperparameter and architecture search for machine learning practitioners, representing an incremental advancement in kernel design for Bayesian optimization.
The paper tackles the problem of Bayesian optimization over structures with varying numbers of parameters, such as neural network architectures, by defining a new kernel for conditional parameter spaces that incorporates parameter relevance information, resulting in improved model quality and optimization outcomes compared to simpler baseline kernels.
In practical Bayesian optimization, we must often search over structures with differing numbers of parameters. For instance, we may wish to search over neural network architectures with an unknown number of layers. To relate performance data gathered for different architectures, we define a new kernel for conditional parameter spaces that explicitly includes information about which parameters are relevant in a given structure. We show that this kernel improves model quality and Bayesian optimization results over several simpler baseline kernels.