Model Directions, Not Words: Mechanistic Topic Models Using Sparse Autoencoders
This addresses the problem of articulating complex topics in text analysis for researchers and practitioners, offering an incremental improvement over existing neural variants.
The paper tackles the limitation of traditional topic models in capturing abstract semantic features by introducing Mechanistic Topic Models (MTMs), which use sparse autoencoders to define topics over interpretable features, resulting in matching or exceeding baselines on coherence metrics across five datasets and enabling controllable text generation.
Traditional topic models are effective at uncovering latent themes in large text collections. However, due to their reliance on bag-of-words representations, they struggle to capture semantically abstract features. While some neural variants use richer representations, they are similarly constrained by expressing topics as word lists, which limits their ability to articulate complex topics. We introduce Mechanistic Topic Models (MTMs), a class of topic models that operate on interpretable features learned by sparse autoencoders (SAEs). By defining topics over this semantically rich space, MTMs can reveal deeper conceptual themes with expressive feature descriptions. Moreover, uniquely among topic models, MTMs enable controllable text generation using topic-based steering vectors. To properly evaluate MTM topics against word-list-based approaches, we propose \textit{topic judge}, an LLM-based pairwise comparison evaluation framework. Across five datasets, MTMs match or exceed traditional and neural baselines on coherence metrics, are consistently preferred by topic judge, and enable effective steering of LLM outputs.