Learning To Retrieve Prompts for In-Context Learning
This addresses the challenge of prompt sensitivity in in-context learning for NLP practitioners, though it is incremental as it builds on existing retrieval and probability estimation techniques.
The paper tackles the problem of selecting effective training examples (prompts) for in-context learning in natural language understanding, proposing a method that retrieves prompts using annotated data and a language model, and reports substantial outperformance over prior work and baselines on three sequence-to-sequence tasks.
In-context learning is a recent paradigm in natural language understanding, where a large pre-trained language model (LM) observes a test instance and a few training examples as its input, and directly decodes the output without any update to its parameters. However, performance has been shown to strongly depend on the selected training examples (termed prompt). In this work, we propose an efficient method for retrieving prompts for in-context learning using annotated data and a LM. Given an input-output pair, we estimate the probability of the output given the input and a candidate training example as the prompt, and label training examples as positive or negative based on this probability. We then train an efficient dense retriever from this data, which is used to retrieve training examples as prompts at test time. We evaluate our approach on three sequence-to-sequence tasks where language utterances are mapped to meaning representations, and find that it substantially outperforms prior work and multiple baselines across the board.