Bilinear autoencoders find interpretable manifolds
For interpretability researchers, this provides a mathematically tractable nonlinear method for discovering interpretable manifolds in neural networks, though it is an incremental step building on sparse autoencoders.
The paper introduces bilinear autoencoders with quadratic latents to capture multi-dimensional manifolds in neural network activations, challenging the linear representation hypothesis. Experiments show these models systematically improve reconstruction error in language models and recover the same input subspace as linear methods, despite distinct dictionary entries.
Sparse autoencoders have become a standard tool for uncovering interpretable latent representations in neural networks. Yet salient concepts often span manifolds that current linear methods cannot capture without post hoc analysis. This paper uses quadratic latents to close this gap: we implement these with bilinear autoencoders, which decompose activations into low-rank quadratic forms, compose linearly in weight space, and admit input-independent geometric analysis. This qualitative difference in what concepts quadratic latents can detect challenges the standard linear representation hypothesis. Our experiments and visualisations show that multi-dimensional geometries are highly prevalent and that composite latents capture them well, systematically improving reconstruction error in language models. Furthermore, we show that autoencoders with varying geometric priors recover the same input subspace despite their dictionary entries being distinct. Practically, these models serve as an unsupervised tool for manifold discovery, which we demonstrate through an interactive online visualizer for Qwen 3.5. This is a step toward nonlinear but mathematically tractable latent representations whose composition is expressive and interpretable by design.