Language Models Grow Less Humanlike beyond Phase Transition
This addresses a fundamental issue in AI alignment for researchers and developers, providing insights into pretraining dynamics that could improve model interpretability and human-like behavior, though it is incremental in explaining specific degradation patterns.
The paper investigates why language models' alignment with human reading behavior improves during pretraining up to a tipping point and then degrades, attributing this to a phase transition characterized by specialized attention heads that alter learning dynamics, leading to continued decline in psychometric predictive power.
LMs' alignment with human reading behavior (i.e. psychometric predictive power; PPP) is known to improve during pretraining up to a tipping point, beyond which it either plateaus or degrades. Various factors, such as word frequency, recency bias in attention, and context size, have been theorized to affect PPP, yet there is no current account that explains why such a tipping point exists, and how it interacts with LMs' pretraining dynamics more generally. We hypothesize that the underlying factor is a pretraining phase transition, characterized by the rapid emergence of specialized attention heads. We conduct a series of correlational and causal experiments to show that such a phase transition is responsible for the tipping point in PPP. We then show that, rather than producing attention patterns that contribute to the degradation in PPP, phase transitions alter the subsequent learning dynamics of the model, such that further training keeps damaging PPP.