Testing the spin-bath view of self-attention: A Hamiltonian analysis of GPT-2 Transformer
This provides the first strong empirical evidence for a physics-based interpretation of attention mechanisms in production-grade language models, potentially bridging condensed matter physics and AI.
The researchers tested whether the attention mechanism in GPT-2 can be modeled as an interacting spin system by extracting Query-Key weight matrices and deriving effective Hamiltonians, finding a strong negative correlation (r≈-0.70, p<10^-3) between theoretical predictions and empirical token rankings across 144 heads.
The recently proposed physics-based framework by Huo and Johnson~\cite{huo2024capturing} models the attention mechanism of Large Language Models (LLMs) as an interacting two-body spin system, offering a first-principles explanation for phenomena like repetition and bias. Building on this hypothesis, we extract the complete Query-Key weight matrices from a production-grade GPT-2 model and derive the corresponding effective Hamiltonian for every attention head. From these Hamiltonians, we obtain analytic phase boundaries and logit gap criteria that predict which token should dominate the next-token distribution for a given context. A systematic evaluation on 144 heads across 20 factual-recall prompts reveals a strong negative correlation between the theoretical logit gaps and the model's empirical token rankings ($r\approx-0.70$, $p<10^{-3}$).Targeted ablations further show that suppressing the heads most aligned with the spin-bath predictions induces the anticipated shifts in output probabilities, confirming a causal link rather than a coincidental association. Taken together, our findings provide the first strong empirical evidence for the spin-bath analogy in a production-grade model. In this work, we utilize the context-field lens, which provides physics-grounded interpretability and motivates the development of novel generative models bridging theoretical condensed matter physics and artificial intelligence.