Sparse Feature Coactivation Reveals Causal Semantic Modules in Large Language Models
This work addresses the challenge of understanding and controlling knowledge organization in LLMs for researchers and practitioners, though it is incremental as it builds on existing methods for feature analysis.
The researchers tackled the problem of identifying and manipulating semantically coherent network components in large language models (LLMs) by using coactivation of sparse autoencoder features from a few prompts, showing that ablating or amplifying these components predictably changes model outputs and induces counterfactual responses, with concept components emerging early and relation components in later layers.
We identify semantically coherent, context-consistent network components in large language models (LLMs) using coactivation of sparse autoencoder (SAE) features collected from just a handful of prompts. Focusing on concept-relation prediction tasks, we show that ablating these components for concepts (e.g., countries and words) and relations (e.g., capital city and translation language) changes model outputs in predictable ways, while amplifying these components induces counterfactual responses. Notably, composing relation and concept components yields compound counterfactual outputs. Further analysis reveals that while most concept components emerge from the very first layer, more abstract relation components are concentrated in later layers. Lastly, we show that extracted components more comprehensively capture concepts and relations than individual features while maintaining specificity. Overall, our findings suggest a modular organization of knowledge accessed through compositional operations, and advance methods for efficient, targeted LLM manipulation.