Understanding In-Context Learning from Repetitions
This provides insights into a fundamental AI capability, but it is incremental as it builds on existing understanding of in-context learning.
The paper tackles the mechanism of in-context learning in Large Language Models by examining surface repetitions, identifying token co-occurrence reinforcement as a key principle that strengthens token relationships based on contextual co-occurrences, and explaining reasons for its failures.
This paper explores the elusive mechanism underpinning in-context learning in Large Language Models (LLMs). Our work provides a novel perspective by examining in-context learning via the lens of surface repetitions. We quantitatively investigate the role of surface features in text generation, and empirically establish the existence of \emph{token co-occurrence reinforcement}, a principle that strengthens the relationship between two tokens based on their contextual co-occurrences. By investigating the dual impacts of these features, our research illuminates the internal workings of in-context learning and expounds on the reasons for its failures. This paper provides an essential contribution to the understanding of in-context learning and its potential limitations, providing a fresh perspective on this exciting capability.