Learning to Search Effective Example Sequences for In-Context Learning
This work addresses the challenge of improving few-shot learning efficiency for users of large language models, though it is incremental as it builds on existing methods by integrating previously isolated factors.
The paper tackles the problem of optimizing in-context example sequences for large language models, where performance varies based on factors like length and arrangement, and introduces BESC to jointly address these factors, resulting in notable performance improvements across datasets and models.
Large language models (LLMs) demonstrate impressive few-shot learning capabilities, but their performance varies widely based on the sequence of in-context examples. Key factors influencing this include the sequence's length, composition, and arrangement, as well as its relation to the specific query. Existing methods often tackle these factors in isolation, overlooking their interdependencies. Moreover, the extensive search space for selecting optimal sequences complicates the development of a holistic approach. In this work, we introduce Beam Search-based Example Sequence Constructor (BESC), a novel method for learning to construct optimal example sequences. BESC addresses all key factors involved in sequence selection by considering them jointly during inference, while incrementally building the sequence. This design enables the use of beam search to significantly reduce the complexity of the search space. Experiments across various datasets and language models show notable improvements in performance.