Enhancing Trust-Region Bayesian Optimization via Newton Methods
This work addresses the problem of high-dimensional optimization for researchers and practitioners in machine learning, offering an incremental improvement over prior trust-region approaches.
The paper tackles the challenge of scaling Bayesian Optimization to high-dimensional spaces by enhancing trust-region methods with Newton techniques, resulting in improved sampling efficiency and outperforming existing methods on synthetic and real-world benchmarks.
Bayesian Optimization (BO) has been widely applied to optimize expensive black-box functions while retaining sample efficiency. However, scaling BO to high-dimensional spaces remains challenging. Existing literature proposes performing standard BO in multiple local trust regions (TuRBO) for heterogeneous modeling of the objective function and avoiding over-exploration. Despite its advantages, using local Gaussian Processes (GPs) reduces sampling efficiency compared to a global GP. To enhance sampling efficiency while preserving heterogeneous modeling, we propose to construct multiple local quadratic models using gradients and Hessians from a global GP, and select new sample points by solving the bound-constrained quadratic program. Additionally, we address the issue of vanishing gradients of GPs in high-dimensional spaces. We provide a convergence analysis and demonstrate through experimental results that our method enhances the efficacy of TuRBO and outperforms a wide range of high-dimensional BO techniques on synthetic functions and real-world applications.