From Directions to Regions: Decomposing Activations in Language Models via Local Geometry
This work addresses the challenge of scalable concept discovery and model control in language models for researchers and practitioners, offering an incremental improvement over existing decomposition methods by accounting for complex, nonlinear activation structures.
The paper tackled the problem of decomposing activations in language models by moving from global directions to local geometric regions, using Mixture of Factor Analyzers to model nonlinear structures, and showed that this approach outperforms unsupervised baselines and is competitive with supervised methods in localization and steering benchmarks.
Activation decomposition methods in language models are tightly coupled to geometric assumptions on how concepts are realized in activation space. Existing approaches search for individual global directions, implicitly assuming linear separability, which overlooks concepts with nonlinear or multi-dimensional structure. In this work, we leverage Mixture of Factor Analyzers (MFA) as a scalable, unsupervised alternative that models the activation space as a collection of Gaussian regions with their local covariance structure. MFA decomposes activations into two compositional geometric objects: the region's centroid in activation space, and the local variation from the centroid. We train large-scale MFAs for Llama-3.1-8B and Gemma-2-2B, and show they capture complex, nonlinear structures in activation space. Moreover, evaluations on localization and steering benchmarks show that MFA outperforms unsupervised baselines, is competitive with supervised localization methods, and often achieves stronger steering performance than sparse autoencoders. Together, our findings position local geometry, expressed through subspaces, as a promising unit of analysis for scalable concept discovery and model control, accounting for complex structures that isolated directions fail to capture.