Batch Acquisition Function Evaluations and Decouple Optimizer Updates for Faster Bayesian Optimization
This work addresses a specific computational inefficiency for users of Bayesian optimization libraries like BoTorch and Optuna, representing an incremental improvement.
The paper tackles the computational bottleneck in Bayesian optimization caused by suboptimal multi-start optimization due to off-diagonal approximation errors in quasi-Newton methods, and proposes a decoupled update approach that reduces wall-clock time while maintaining theoretical convergence.
Bayesian optimization (BO) efficiently finds high-performing parameters by maximizing an acquisition function, which models the promise of parameters. A major computational bottleneck arises in acquisition function optimization, where multi-start optimization (MSO) with quasi-Newton (QN) methods is required due to the non-convexity of the acquisition function. BoTorch, a widely used BO library, currently optimizes the summed acquisition function over multiple points, leading to the speedup of MSO owing to PyTorch batching. Nevertheless, this paper empirically demonstrates the suboptimality of this approach in terms of off-diagonal approximation errors in the inverse Hessian of a QN method, slowing down its convergence. To address this problem, we propose to decouple QN updates using a coroutine while batching the acquisition function calls. Our approach not only yields the theoretically identical convergence to the sequential MSO but also drastically reduces the wall-clock time compared to the previous approaches. Our approach is available in GPSampler in Optuna, effectively reducing its computational overhead.