Repetitions are not all alike: distinct mechanisms sustain repetition in language models
This work addresses the problem of understanding and mitigating repetitive failures in LLMs for AI researchers and developers, though it is incremental as it builds on existing knowledge of model behavior.
The study investigated whether repetitive loops in Large Language Models arise from distinct mechanisms, finding that in-context learning-induced repetition involves specialized attention heads developed during training, while natural repetition emerges early and focuses on low-information tokens as a fallback behavior.
Large Language Models (LLMs) can sometimes degrade into repetitive loops, persistently generating identical word sequences. Because repetition is rare in natural human language, its frequent occurrence across diverse tasks and contexts in LLMs remains puzzling. Here we investigate whether behaviorally similar repetition patterns arise from distinct underlying mechanisms and how these mechanisms develop during model training. We contrast two conditions: repetitions elicited by natural text prompts with those induced by in-context learning (ICL) setups that explicitly require copying behavior. Our analyses reveal that ICL-induced repetition relies on a dedicated network of attention heads that progressively specialize over training, whereas naturally occurring repetition emerges early and lacks a defined circuitry. Attention inspection further shows that natural repetition focuses disproportionately on low-information tokens, suggesting a fallback behavior when relevant context cannot be retrieved. These results indicate that superficially similar repetition behaviors originate from qualitatively different internal processes, reflecting distinct modes of failure and adaptation in language models.