Monte Carlo Sampling for Analyzing In-Context Examples
This work addresses the problem of optimizing in-context learning for AI practitioners, but it is incremental as it builds on prior studies of presentation factors without introducing a new paradigm.
The paper investigates the brittleness of in-context learning to presentation factors like example order and selection, using a Monte Carlo sampling method to analyze the impact of example number while accounting for these factors. It finds that prior guidance on example selection does not generalize well, and that selecting examples based on data valuation leads to unexpected performance degradation compared to random sampling.
Prior works have shown that in-context learning is brittle to presentation factors such as the order, number, and choice of selected examples. However, ablation-based guidance on selecting the number of examples may ignore the interplay between different presentation factors. In this work we develop a Monte Carlo sampling-based method to study the impact of number of examples while explicitly accounting for effects from order and selected examples. We find that previous guidance on how many in-context examples to select does not always generalize across different sets of selected examples and orderings, and whether one-shot settings outperform zero-shot settings is highly dependent on the selected example. Additionally, inspired by data valuation, we apply our sampling method to in-context example selection to select examples that perform well across different orderings. We find a negative result, that while performance is robust to ordering and number of examples, there is an unexpected performance degradation compared to random sampling.