Cross-seed explainability using Procrustes-conditioned Joint End-to-end Top-K Sparse Autoencoders
For mechanistic interpretability researchers, this method improves cross-seed feature universality, addressing a fundamental challenge in dictionary learning for neural networks.
The paper introduces a Procrustes-conditioned Joint End-to-end Top-K Sparse Autoencoder to extract cross-seed universal features from independently trained BERT models, achieving Pearson r ≥ 0.70 across seeds on three benchmark datasets, outperforming post-hoc alignment baselines.
We present a Procrustes-conditioned Joint End-to-end Top-K Sparse Autoencoder (SAE) for extracting cross-seed universal features from independently trained BERT models. Cross-seed feature universality is a fundamental challenge in mechanistic interpretability: because dictionary learning is non-convex, independently trained networks learn misaligned feature spaces, so apparently identical features may differ by random initialization. We address this by computing an orthogonal Procrustes rotation between seeds' activation spaces before joint SAE training, combining Top-K sparsity, end-to-end downstream optimization, and an auxiliary dead-feature revival loss based on previous SAE literature. Evaluating on five independent seed pairs (ten BERT models) across three benchmark datasets (SST-2, Stanford Politeness, TweetEval Emotion), our full pipeline produces more universal features (Pearson r $\geq$ 0.70 across seeds) than post-hoc alignment baselines on all three datasets. A minimal qualitative analysis confirms that high-universality features encode interpretable sociolinguistic patterns.