Large Scale Multi-Task Bayesian Optimization with Large Language Models
This addresses the scalability bottleneck in multi-task Bayesian optimization for applications like database query optimization and antimicrobial peptide design, representing a novel method rather than an incremental improvement.
The paper tackles the problem of scaling multi-task Bayesian optimization beyond a moderate number of tasks by leveraging large language models (LLMs) to learn from previous optimization trajectories, achieving performance on approximately 1500 tasks where the LLM generates solutions better than from-scratch Bayesian optimization with fewer oracle calls.
In multi-task Bayesian optimization, the goal is to leverage experience from optimizing existing tasks to improve the efficiency of optimizing new ones. While approaches using multi-task Gaussian processes or deep kernel transfer exist, the performance improvement is marginal when scaling beyond a moderate number of tasks. We introduce a novel approach leveraging large language models (LLMs) to learn from, and improve upon, previous optimization trajectories, scaling to approximately 1500 distinct tasks. Specifically, we propose a feedback loop in which an LLM is fine-tuned on the high quality solutions to specific tasks found by Bayesian optimization (BO). This LLM is then used to generate initialization points for future BO searches for new tasks. The trajectories of these new searches provide additional training data for fine-tuning the LLM, completing the loop. We evaluate our method on two distinct domains: database query optimization and antimicrobial peptide design. Results demonstrate that our approach creates a positive feedback loop, where the LLM's generated initializations gradually improve, leading to better optimization performance. As this feedback loop continues, we find that the LLM is eventually able to generate solutions to new tasks in just a few shots that are better than the solutions produced by "from scratch" by Bayesian optimization while simultaneously requiring significantly fewer oracle calls.