From Atoms to Trees: Building a Structured Feature Forest with Hierarchical Sparse Autoencoders
This work provides a scalable tool for discovering multi-scale conceptual structures in LLM representations, addressing a domain-specific need in interpretability research.
The paper tackled the problem of extracting isolated monosemantic features from large language models by proposing a Hierarchical Sparse Autoencoder (HSAE) that learns hierarchical parent-child relationships between features, resulting in consistent recovery of semantically meaningful hierarchies across various models and layers while preserving reconstruction fidelity and interpretability.
Sparse autoencoders (SAEs) have proven effective for extracting monosemantic features from large language models (LLMs), yet these features are typically identified in isolation. However, broad evidence suggests that LLMs capture the intrinsic structure of natural language, where the phenomenon of "feature splitting" in particular indicates that such structure is hierarchical. To capture this, we propose the Hierarchical Sparse Autoencoder (HSAE), which jointly learns a series of SAEs and the parent-child relationships between their features. HSAE strengthens the alignment between parent and child features through two novel mechanisms: a structural constraint loss and a random feature perturbation mechanism. Extensive experiments across various LLMs and layers demonstrate that HSAE consistently recovers semantically meaningful hierarchies, supported by both qualitative case studies and rigorous quantitative metrics. At the same time, HSAE preserves the reconstruction fidelity and interpretability of standard SAEs across different dictionary sizes. Our work provides a powerful, scalable tool for discovering and analyzing the multi-scale conceptual structures embedded in LLM representations.