Fresh in memory: Training-order recency is linearly encoded in language model activations
This reveals how models manage conflicting data and knowledge modifications, with implications for AI interpretability and robustness, though it is incremental as it builds on existing fine-tuning and probing techniques.
The paper tackled the problem of whether language models encode the temporal order of learned information during training, finding that activations linearly encode training recency with centroids arranged in order on a straight line and achieving ~90% accuracy in distinguishing early vs. late entities.
We show that language models' activations linearly encode when information was learned during training. Our setup involves creating a model with a known training order by sequentially fine-tuning Llama-3.2-1B on six disjoint but otherwise similar datasets about named entities. We find that the average activations of test samples corresponding to the six training datasets encode the training order: when projected into a 2D subspace, these centroids are arranged exactly in the order of training and lie on a straight line. Further, we show that linear probes can accurately (~90%) distinguish "early" vs. "late" entities, generalizing to entities unseen during the probes' own training. The model can also be fine-tuned to explicitly report an unseen entity's training stage (~80% accuracy). Interestingly, the training-order encoding does not seem attributable to simple differences in activation magnitudes, losses, or model confidence. Our paper demonstrates that models are capable of differentiating information by its acquisition time, and carries significant implications for how they might manage conflicting data and respond to knowledge modifications.