Hyperband-based Bayesian Optimization for Black-box Prompt Selection
This addresses the challenge of prompt selection for users relying on API-based LLMs, offering a more efficient method, though it is incremental as it builds on existing optimization techniques.
The paper tackles the problem of efficiently selecting optimal prompts for large language models in black-box settings, proposing HbBoPs, which combines a structural-aware deep kernel Gaussian Process with Hyperband to outperform state-of-the-art methods across ten benchmarks and three LLMs in performance and efficiency.
Optimal prompt selection is crucial for maximizing large language model (LLM) performance on downstream tasks, especially in black-box settings where models are only accessible via APIs. Black-box prompt selection is challenging due to potentially large, combinatorial search spaces, absence of gradient information, and high evaluation cost of prompts on a validation set. We propose HbBoPs, a novel method that combines a structural-aware deep kernel Gaussian Process with Hyperband as a multi-fidelity scheduler to efficiently select prompts. HbBoPs uses embeddings of instructions and few-shot exemplars, treating them as modular components within prompts. This enhances the surrogate model's ability to predict which prompt to evaluate next in a sample-efficient manner. Hyperband improves query-efficiency by adaptively allocating resources across different fidelity levels, reducing the number of validation instances required for evaluating prompts. Extensive experiments across ten diverse benchmarks and three LLMs demonstrate that HbBoPs outperforms state-of-the-art methods in both performance and efficiency.