Automatic Prompt Selection for Large Language Models
This work addresses the problem of inefficient prompt design for LLM users, offering a flexible and efficient method, though it is incremental as it builds on existing prompt optimization techniques.
The paper tackles the challenge of manually designing effective prompts for large language models by proposing an automatic prompt selection approach that clusters training data, generates candidate prompts, and uses a prompt evaluator to select the optimal prompt for new inputs, achieving competitive performance on zero-shot question-answering datasets like GSM8K, MultiArith, and AQuA.
Large Language Models (LLMs) can perform various natural language processing tasks with suitable instruction prompts. However, designing effective prompts manually is challenging and time-consuming. Existing methods for automatic prompt optimization either lack flexibility or efficiency. In this paper, we propose an effective approach to automatically select the optimal prompt for a given input from a finite set of synthetic candidate prompts. Our approach consists of three steps: (1) clustering the training data and generating candidate prompts for each cluster using an LLM-based prompt generator; (2) synthesizing a dataset of input-prompt-output tuples for training a prompt evaluator to rank the prompts based on their relevance to the input; (3) using the prompt evaluator to select the best prompt for a new input at test time. Our approach balances prompt generality-specificity and eliminates the need for resource-intensive training and inference. It demonstrates competitive performance on zero-shot question-answering datasets: GSM8K, MultiArith, and AQuA.