Priors in Time: Missing Inductive Biases for Language Model Interpretability
This addresses the need for interpretability tools that match the temporal structure of language, offering a more robust method for researchers and practitioners in AI interpretability.
The paper tackled the problem of recovering meaningful concepts from language model activations by showing that existing Sparse Autoencoders (SAEs) impose priors assuming independence across time, which conflicts with the temporal dynamics in language. They introduced Temporal Feature Analysis, which correctly parses garden path sentences and identifies event boundaries, outperforming SAEs in these tasks.
Recovering meaningful concepts from language model activations is a central aim of interpretability. While existing feature extraction methods aim to identify concepts that are independent directions, it is unclear if this assumption can capture the rich temporal structure of language. Specifically, via a Bayesian lens, we demonstrate that Sparse Autoencoders (SAEs) impose priors that assume independence of concepts across time, implying stationarity. Meanwhile, language model representations exhibit rich temporal dynamics, including systematic growth in conceptual dimensionality, context-dependent correlations, and pronounced non-stationarity, in direct conflict with the priors of SAEs. Taking inspiration from computational neuroscience, we introduce a new interpretability objective -- Temporal Feature Analysis -- which possesses a temporal inductive bias to decompose representations at a given time into two parts: a predictable component, which can be inferred from the context, and a residual component, which captures novel information unexplained by the context. Temporal Feature Analyzers correctly parse garden path sentences, identify event boundaries, and more broadly delineate abstract, slow-moving information from novel, fast-moving information, while existing SAEs show significant pitfalls in all the above tasks. Overall, our results underscore the need for inductive biases that match the data in designing robust interpretability tools.