Why is prompting hard? Understanding prompts on binary sequence predictors
This work addresses the challenge of effective prompting for LLM users, but it is incremental as it builds on existing statistical perspectives.
The paper investigates the difficulty of finding and understanding optimal prompts for large language models, showing that unintuitive patterns in optimal prompts relate to the pretraining distribution and that common prompting methods are suboptimal.
Large language models (LLMs) can be prompted to do many tasks, but finding good prompts is not always easy, nor is understanding some performant prompts. We explore these issues by viewing prompting as conditioning a near-optimal sequence predictor (LLM) pretrained on diverse data sources. Through numerous prompt search experiments, we show that the unintuitive patterns in optimal prompts can be better understood given the pretraining distribution, which is often unavailable in practice. Moreover, even using exhaustive search, reliably identifying optimal prompts from practical neural predictors can be difficult. Further, we demonstrate that common prompting methods, such as using intuitive prompts or samples from the targeted task, are in fact suboptimal. Thus, this work takes an initial step towards understanding the difficulties in finding and understanding optimal prompts from a statistical and empirical perspective.