Adaptive Kernel Design for Bayesian Optimization Is a Piece of CAKE with LLMs
This work addresses a key bottleneck in Bayesian optimization for practitioners in fields such as machine learning and engineering, offering a novel approach to kernel design, though it is incremental in its application of LLMs to an existing framework.
The paper tackles the problem of inefficient kernel selection in Bayesian optimization by proposing CAKE, a method that uses large language models to adaptively generate and refine Gaussian process kernels, resulting in consistent performance improvements across real-world tasks like hyperparameter optimization and photonic chip design.
The efficiency of Bayesian optimization (BO) relies heavily on the choice of the Gaussian process (GP) kernel, which plays a central role in balancing exploration and exploitation under limited evaluation budgets. Traditional BO methods often rely on fixed or heuristic kernel selection strategies, which can result in slow convergence or suboptimal solutions when the chosen kernel is poorly suited to the underlying objective function. To address this limitation, we propose a freshly-baked Context-Aware Kernel Evolution (CAKE) to enhance BO with large language models (LLMs). Concretely, CAKE leverages LLMs as the crossover and mutation operators to adaptively generate and refine GP kernels based on the observed data throughout the optimization process. To maximize the power of CAKE, we further propose BIC-Acquisition Kernel Ranking (BAKER) to select the most effective kernel through balancing the model fit measured by the Bayesian information criterion (BIC) with the expected improvement at each iteration of BO. Extensive experiments demonstrate that our fresh CAKE-based BO method consistently outperforms established baselines across a range of real-world tasks, including hyperparameter optimization, controller tuning, and photonic chip design. Our code is publicly available at https://github.com/richardcsuwandi/cake.