Finding Support Examples for In-Context Learning
This addresses the challenge of efficiently finding support examples for in-context learning in language models, which is an incremental advancement in optimizing example selection.
The paper tackles the NP-hard combinatorial optimization problem of selecting effective in-context examples for language models by proposing LENS, a two-stage method that filters examples using a novel InfoScore metric and searches for diverse permutations, resulting in significant performance improvements over baselines.
Additionally, the strong dependency among in-context examples makes it an NP-hard combinatorial optimization problem and enumerating all permutations is infeasible. Hence we propose LENS, a fiLter-thEN-Search method to tackle this challenge in two stages: First we filter the dataset to obtain informative in-context examples individually. Specifically, we propose a novel metric, InfoScore, to evaluate the example's in-context informativeness based on the language model's feedback, and further propose a progressive filtering process to filter out uninformative examples. Then we propose diversity-guided example search which iteratively refines and evaluates the selected example permutations, to find examples that fully depict the task. The experimental results show that LENS significantly outperforms a wide range of baselines.