In-context Example Selection with Influences
This addresses the problem of optimizing few-shot in-context learning for users of large language models, though it is incremental as it builds on existing influence analysis techniques.
The paper tackles the sensitivity of in-context learning performance to input examples by proposing an influence-based method for selecting positive and negative examples, achieving up to a 16.3% performance gap improvement on 9 SuperGLUE tasks.
In-context learning (ICL) is a powerful paradigm emerged from large language models (LLMs). Despite its promises, ICL performance is known to be highly sensitive to input examples. In this work, we use $\textit{in-context influences}$ to analyze few-shot ICL performance directly from the in-context examples. Our proposed influence-based example selection method can identify both positive and negative examples, outperforming several baselines when evaluated on 9 SuperGLUE tasks. Our analysis uncovers up to a $16.3\%$ performance gap between using the most negative in-context examples compared to the most positive. In a case study, we apply our influence-based framework to quantify the phenomena of recency bias in example ordering for few-shot ICL.