Hidden Breakthroughs in Language Model Training
This work addresses the challenge of unsupervised interpretability in AI training for researchers, though it is incremental as it builds on existing methods for analyzing learning dynamics.
The paper tackles the problem of identifying hidden conceptual breakthroughs during language model training by introducing POLCA, a method that decomposes loss changes to reveal clusters of data with similar learning dynamics, validated on synthetic and natural language tasks.
Loss curves are smooth during most of model training, so visible discontinuities stand out as possible conceptual breakthroughs. Studying these breakthroughs enables a deeper understanding of learning dynamics, but only when they are properly identified. This paper argues that similar breakthroughs occur frequently throughout training but they are obscured by a loss metric that collapses all variation into a single scalar. To find these hidden transitions, we introduce POLCA, a method for decomposing changes in loss along arbitrary bases of the low-rank training subspace. We use our method to identify clusters of samples that share similar changes in loss during training, disaggregating the overall loss into that of smaller groups of conceptually similar data. We validate our method on synthetic arithmetic and natural language tasks, showing that POLCA recovers clusters that represent interpretable breakthroughs in the model's capabilities. We demonstrate the promise of these hidden phase transitions as a tool for unsupervised interpretability.