Incorporating Hierarchical Semantics in Sparse Autoencoder Architectures
This addresses the problem of enhancing interpretability and efficiency in sparse autoencoders for researchers and practitioners in machine learning, though it appears incremental as it builds on existing SAE methods.
The paper tackled the limitation of sparse autoencoders not representing semantic relationships between learned concepts by introducing a modified architecture that explicitly models a semantic hierarchy. The result showed that this approach improves reconstruction, interpretability, and computational efficiency when applied to large language models.
Sparse dictionary learning (and, in particular, sparse autoencoders) attempts to learn a set of human-understandable concepts that can explain variation on an abstract space. A basic limitation of this approach is that it neither exploits nor represents the semantic relationships between the learned concepts. In this paper, we introduce a modified SAE architecture that explicitly models a semantic hierarchy of concepts. Application of this architecture to the internal representations of large language models shows both that semantic hierarchy can be learned, and that doing so improves both reconstruction and interpretability. Additionally, the architecture leads to significant improvements in computational efficiency.