In-Context Exemplars as Clues to Retrieving from Large Associative Memory
This work provides a novel perspective on ICL mechanisms, potentially advancing understanding of LLMs, but it is incremental as it builds on existing ICL and memory theories without broad SOTA impact.
The paper tackles the problem of how in-context exemplars affect in-context learning (ICL) in large language models by conceptualizing ICL as retrieval from associative memory, establishing a theoretical framework based on Hopfield Networks and proposing more efficient active exemplar selection methods.
Recently, large language models (LLMs) have made remarkable progress in natural language processing. The most representative ability of LLMs is in-context learning (ICL), which enables LLMs to learn patterns from in-context exemplars without training. The performance of ICL greatly depends on the exemplars used. However, how to choose exemplars remains unclear due to the lack of understanding of how in-context learning works. In this paper, we present a novel perspective on ICL by conceptualizing it as contextual retrieval from a model of associative memory. We establish a theoretical framework of ICL based on Hopfield Networks. Based on our framework, we look into how in-context exemplars influence the performance of ICL and propose more efficient active exemplar selection. Our study sheds new light on the mechanism of ICL by connecting it to memory retrieval, with potential implications for advancing the understanding of LLMs.