Provable Low-Frequency Bias of In-Context Learning of Representations
This provides a theoretical foundation for understanding ICL mechanisms, addressing an open question in AI research, though it is incremental as it builds on prior empirical observations.
The paper tackles the problem of explaining how in-context learning (ICL) in large language models internalizes data structures, proving that it leads to an implicit bias towards smooth (low-frequency) representations through a double convergence framework, and empirically verifying this bias and its robustness to high-frequency noise.
In-context learning (ICL) enables large language models (LLMs) to acquire new behaviors from the input sequence alone without any parameter updates. Recent studies have shown that ICL can surpass the original meaning learned in pretraining stage through internalizing the structure the data-generating process (DGP) of the prompt into the hidden representations. However, the mechanisms by which LLMs achieve this ability is left open. In this paper, we present the first rigorous explanation of such phenomena by introducing a unified framework of double convergence, where hidden representations converge both over context and across layers. This double convergence process leads to an implicit bias towards smooth (low-frequency) representations, which we prove analytically and verify empirically. Our theory explains several open empirical observations, including why learned representations exhibit globally structured but locally distorted geometry, and why their total energy decays without vanishing. Moreover, our theory predicts that ICL has an intrinsic robustness towards high-frequency noise, which we empirically confirm. These results provide new insights into the underlying mechanisms of ICL, and a theoretical foundation to study it that hopefully extends to more general data distributions and settings.