From Isolation to Entanglement: When Do Interpretability Methods Identify and Disentangle Known Concepts?
This addresses the problem of evaluating concept disentanglement in interpretability for AI researchers, highlighting limitations in current methods and emphasizing the need for compositional evaluations.
The study investigated whether common interpretability methods like sparse autoencoders and sparse probes recover disentangled representations of concepts in neural networks, finding that features correspond to no more than one concept but concepts are distributed across many features, and steering experiments showed features affect multiple concepts, indicating lack of selectivity and independence.
A central goal of interpretability is to recover representations of causally relevant concepts from the activations of neural networks. The quality of these concept representations is typically evaluated in isolation, and under implicit independence assumptions that may not hold in practice. Thus, it is unclear whether common featurization methods - including sparse autoencoders (SAEs) and sparse probes - recover disentangled representations of these concepts. This study proposes a multi-concept evaluation setting where we control the correlations between textual concepts, such as sentiment, domain, and tense, and analyze performance under increasing correlations between them. We first evaluate the extent to which featurizers can learn disentangled representations of each concept under increasing correlational strengths. We observe a one-to-many relationship from concepts to features: features correspond to no more than one concept, but concepts are distributed across many features. Then, we perform steering experiments, measuring whether each concept is independently manipulable. Even when trained on uniform distributions of concepts, SAE features generally affect many concepts when steered, indicating that they are neither selective nor independent; nonetheless, features affect disjoint subspaces. These results suggest that correlational metrics for measuring disentanglement are generally not sufficient for establishing independence when steering, and that affecting disjoint subspaces is not sufficient for concept selectivity. These results underscore the importance of compositional evaluations in interpretability research.