Predicting the Emergence of Induction Heads in Language Model Pretraining
This work addresses a key bottleneck in understanding in-context learning for language models, though it is incremental in refining existing theories.
The study tackled the problem of predicting when induction heads form during language model pretraining by analyzing statistical properties of training data, showing that a simple equation based on batch and context size predicts emergence and that bigram repetition frequency and reliability strongly influence formation.
Specialized attention heads dubbed induction heads (IHs) have been argued to underlie the remarkable in-context learning capabilities of modern language models; yet, a precise characterization of their emergence, especially in the context of language modeling, remains wanting. In this study, we investigate the relationship between statistical properties of the training data and IH formation in both natural and synthetic training data settings. We show that: (1) A simple equation combining batch size and context size predicts the point at which IHs form and that this emergence point is agnostic to model size; (2) Surface bigram repetition frequency and reliability strongly affect the formation of IHs, and we find an effective Pareto frontier in terms of these two values; (3) local dependency with high bigram repetition frequency and reliability is sufficient for IH formation, but when the frequency and reliability are low, categoriality and the shape of the marginal distribution matter.