TRACE: Training and Inference-Time Interpretability Analysis for Language Models
This provides a more interpretable and actionable tool for researchers studying transformer models, though it is incremental as it builds on existing interpretability methods.
The authors tackled the challenge of analyzing linguistic knowledge emergence in language models by introducing TRACE, a modular toolkit for training and inference-time interpretability analysis, which revealed developmental phenomena like early syntactic emergence and delayed semantic acquisition in autoregressive transformers.
Understanding when and how linguistic knowledge emerges during language model training remains a central challenge for interpretability. Most existing tools are post hoc, rely on scalar metrics, or require nontrivial integration effort, making comprehensive interpretability analysis difficult to deploy and maintain. We introduce TRACE, a modular toolkit for training and inference-time interpretability analysis of transformer models. It enables lightweight, in-training analysis of linguistic and representational signals, including features probing, intrinsic dimensionality, Hessian curvature, and output diagnostics. It integrates with ABSynth, a controllable synthetic corpus generator that provides structured annotations for precise evaluation of linguistic feature acquisition. Experiments with autoregressive transformers demonstrate that TRACE reveals developmental phenomena such as early syntactic emergence, delayed semantic acquisition, and representational compression, signals overlooked by traditional scalar metrics such as loss or accuracy. With minimal integration effort, the tool enables layer-wise diagnostics, convergence-based early stopping, and detection of structural errors, making transformer analysis interpretable, actionable, and reproducible.