Binary Sparse Coding for Interpretability
This work addresses interpretability for neural network researchers, but it is incremental as it builds on existing sparse autoencoder methods.
The authors tackled the problem of interpretability in sparse autoencoders by proposing binary sparse autoencoders and transcoders, which constrain activations to zero or one, finding that binarisation improves interpretability and monosemanticity but increases reconstruction error and ultra-high frequency features.
Sparse autoencoders (SAEs) are used to decompose neural network activations into sparsely activating features, but many SAE features are only interpretable at high activation strengths. To address this issue we propose to use binary sparse autoencoders (BAEs) and binary transcoders (BTCs), which constrain all activations to be zero or one. We find that binarisation significantly improves the interpretability and monosemanticity of the discovered features, while increasing reconstruction error. By eliminating the distinction between high and low activation strengths, we prevent uninterpretable information from being smuggled in through the continuous variation in feature activations. However, we also find that binarisation increases the number of uninterpretable ultra-high frequency features, and when interpretability scores are frequency-adjusted, the scores for continuous sparse coders are slightly better than those of binary ones. This suggests that polysemanticity may be an ineliminable property of neural activations.