Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Ordering
This addresses the inefficiency of random example selection in ICL for NLP tasks, offering a method to enhance few-shot performance, though it is incremental as it builds on existing ICL frameworks.
The paper tackles the problem of randomly selecting and ordering in-context examples in in-context learning (ICL) by proposing a self-adaptive mechanism that finds optimal permutations for each sample, achieving a 40% relative improvement over common practice on eight NLP datasets.
Despite the surprising few-shot performance of in-context learning (ICL), it is still a common practice to randomly sample examples to serve as context. This paper advocates a new principle for ICL: self-adaptive in-context learning. The self-adaption mechanism is introduced to help each sample find an in-context example permutation (i.e., selection and ordering) that can derive the correct prediction, thus maximizing performance. To validate the effectiveness of self-adaptive ICL, we propose a general select-then-rank framework and instantiate it with new selection and ranking algorithms. Upon extensive evaluation on eight different NLP datasets, our self-adaptive ICL method achieves a 40% relative improvement over the common practice setting. Further analysis reveals the enormous potential of self-adaptive ICL that it might be able to close the gap between ICL and finetuning given more advanced algorithms. Our code is released to facilitate future research in this area: https://github.com/Shark-NLP/self-adaptive-ICL