Temporal Dependencies in In-Context Learning: The Role of Induction Heads
This work addresses a fundamental gap in understanding temporal dependencies in in-context learning for AI researchers, though it is incremental as it builds on known mechanisms like induction heads.
The study tackled the problem of how large language models track and retrieve information from context by showing that they exhibit a serial-recall-like pattern, with induction heads playing a key role; ablation experiments revealed that removing high-induction heads reduced the +1 lag bias and impaired serial recall performance.
Large language models (LLMs) exhibit strong in-context learning capabilities, but how they track and retrieve information from context remains underexplored. Drawing on the free recall paradigm in cognitive science (where participants recall list items in any order), we show that several open-source LLMs consistently display a serial-recall-like pattern, assigning peak probability to tokens that immediately follow a repeated token in the input sequence. Through systematic ablation experiments, we show that induction heads, specialized attention heads that attend to the token following a previous occurrence of the current token, play an important role in this phenomenon. Removing heads with a high induction score substantially reduces the +1 lag bias, whereas ablating random heads does not reproduce the same reduction. We also show that removing heads with high induction scores impairs the performance of models prompted to do serial recall using few-shot learning to a larger extent than removing random heads. Our findings highlight a mechanistically specific connection between induction heads and temporal context processing in transformers, suggesting that these heads are especially important for ordered retrieval and serial-recall-like behavior during in-context learning.