Thompson Sampling via Fine-Tuning of LLMs
This addresses scalable Bayesian optimization for researchers and practitioners in fields like bioinformatics and quantum computing, though it appears incremental as it adapts existing Thompson sampling with LLM fine-tuning.
The paper tackles Bayesian optimization in large unstructured discrete spaces by proposing Thompson Sampling via Fine-Tuning (ToSFiT), which eliminates acquisition function maximization by parameterizing the probability of maximum reward. The method improves sample efficiency with negligible computational impact, validated on tasks like FAQ response refinement, protein search, and quantum circuit design.
Bayesian optimization in large unstructured discrete spaces is often hindered by the computational cost of maximizing acquisition functions due to the absence of gradients. We propose a scalable alternative based on Thompson sampling that eliminates the need for acquisition function maximization by directly parameterizing the probability that a candidate yields the maximum reward. Our approach, Thompson Sampling via Fine-Tuning (ToSFiT) leverages the prior knowledge embedded in prompt-conditioned large language models, and incrementally adapts them toward the posterior. Theoretically, we derive a novel regret bound for a variational formulation of Thompson Sampling that matches the strong guarantees of its standard counterpart. Our analysis reveals the critical role of careful adaptation to the posterior probability of maximality--a principle that underpins our ToSFiT algorithm. Empirically, we validate our method on three diverse tasks: FAQ response refinement, thermally stable protein search, and quantum circuit design. We demonstrate that online fine-tuning significantly improves sample efficiency, with negligible impact on computational efficiency.