One size doesn't fit all: Predicting the Number of Examples for In-Context Learning
This addresses the inefficiency of one-size-fits-all ICL for users of large language models, though it is an incremental improvement over existing methods.
The paper tackles the problem of using a fixed number of examples for in-context learning (ICL) in LLMs by dynamically predicting the number of examples per data instance, resulting in up to 17% performance improvement over standard ICL on text classification benchmarks.
In-context learning (ICL) refers to the process of adding a small number of localized examples from a training set of labelled data to an LLM's prompt with an objective to effectively control the generative process seeking to improve the downstream task performance. Existing ICL approaches use an identical number of examples (a pre-configured hyper-parameter) for each data instance. Our work alleviates the limitations of this 'one fits all' approach by dynamically predicting the number of examples for each data instance to be used in few-shot inference with LLMs. In particular, we employ a multi-label classifier, the parameters of which are fitted using a training set, where the label for each instance in this training set indicates if using a specific value of k (number of most similar examples from 0 up to a maximum value) leads to correct k-shot downstream predictions. Our experiments on a number of text classification benchmarks show that AICL substantially outperforms standard ICL by up to 17%.