Kinetic theory for Transformers and the lost-in-the-middle phenomenon
It offers a mathematical foundation for a widely observed empirical phenomenon in large language models, addressing a key limitation in transformer-based architectures.
The paper provides a rigorous theoretical explanation for the lost-in-the-middle phenomenon in decoder Transformers, showing that the token retrieval profile is U-shaped with primacy, recency, and a unique interior minimum under an explicit smallness condition.
We study causal self-attention dynamics -- a toy model for decoder Transformers -- which we interpret as a non-exchangeable interacting particle system. Adapting cumulant expansions to the triangular causal dependency structure of the model, and appealing to non-hierarchical methods to estimate correlations using Glauber calculus, we prove a quantitative mean-field limit result and a next-order characterization of correlations. For iid uniformly distributed tokens, the limiting correlation equation can be solved in closed form and we obtain a rigorous explanation of the empirically observed \emph{lost-in-the-middle} phenomenon: the token retrieval profile, as a function of the source position in the prompt, is $\mathsf{U}$-shaped, with primacy, recency, and a unique interior minimum under an explicit smallness condition.