The Dual-Route Model of Induction
This work addresses the problem of understanding how language models process and represent abstract concepts, offering insights into their internal mechanisms for semantic tasks, though it is incremental in building on prior induction head research.
The paper discovered concept-level induction heads in language models, which copy entire lexical units rather than individual tokens, and showed that these heads enable semantic tasks like word-level translation while token-level induction heads handle verbatim copying, with ablation experiments revealing independent operation and language-independent word representations.
Prior work on in-context copying has shown the existence of induction heads, which attend to and promote individual tokens during copying. In this work we discover a new type of induction head: concept-level induction heads, which copy entire lexical units instead of individual tokens. Concept induction heads learn to attend to the ends of multi-token words throughout training, working in parallel with token-level induction heads to copy meaningful text. We show that these heads are responsible for semantic tasks like word-level translation, whereas token induction heads are vital for tasks that can only be done verbatim (like copying nonsense tokens). These two "routes" operate independently: we show that ablation of token induction heads causes models to paraphrase where they would otherwise copy verbatim. By patching concept induction head outputs, we find that they contain language-independent word representations that mediate natural language translation, suggesting that LLMs represent abstract word meanings independent of language or form.