Tracking the Feature Dynamics in LLM Training: A Mechanistic Study
This work addresses the mechanistic interpretability of LLMs, offering incremental improvements in tracking feature dynamics for researchers in AI interpretability.
The study tackled the problem of understanding how features evolve during large language model (LLM) training by introducing SAE-Track, a method for efficiently tracking sparse autoencoders, and provided new insights into semantic evolution, feature formation, and directional drift.
Understanding training dynamics and feature evolution is crucial for the mechanistic interpretability of large language models (LLMs). Although sparse autoencoders (SAEs) have been used to identify features within LLMs, a clear picture of how these features evolve during training remains elusive. In this study, we (1) introduce SAE-Track, a novel method for efficiently obtaining a continual series of SAEs, providing the foundation for a mechanistic study that covers (2) the semantic evolution of features, (3) the underlying processes of feature formation, and (4) the directional drift of feature vectors. Our work provides new insights into the dynamics of features in LLMs, enhancing our understanding of training mechanisms and feature evolution. For reproducibility, our code is available at https://github.com/Superposition09m/SAE-Track.