Sequential Subspace Search for Functional Bayesian Optimization Incorporating Experimenter Intuition
This work addresses functional optimization in domains like materials science and deep learning, but it is incremental as it builds on standard Bayesian optimization with subspace methods.
The paper tackles the problem of Bayesian functional optimization by incorporating experimenter intuition into the covariance function of a Gaussian Process, proposing an algorithm that uses sequential subspace search and achieves convergence in sub-linear time with a regret bound.
We propose an algorithm for Bayesian functional optimisation - that is, finding the function to optimise a process - guided by experimenter beliefs and intuitions regarding the expected characteristics (length-scale, smoothness, cyclicity etc.) of the optimal solution encoded into the covariance function of a Gaussian Process. Our algorithm generates a sequence of finite-dimensional random subspaces of functional space spanned by a set of draws from the experimenter's Gaussian Process. Standard Bayesian optimisation is applied on each subspace, and the best solution found used as a starting point (origin) for the next subspace. Using the concept of effective dimensionality, we analyse the convergence of our algorithm and provide a regret bound to show that our algorithm converges in sub-linear time provided a finite effective dimension exists. We test our algorithm in simulated and real-world experiments, namely blind function matching, finding the optimal precipitation-strengthening function for an aluminium alloy, and learning rate schedule optimisation for deep networks.