Finding Manifolds With Bilinear Autoencoders
This work addresses the need for nonlinear yet analysable latents in machine learning, though it is an initial step and incremental in nature.
The paper tackles the problem of interpretable latent representations in neural networks by using bilinear autoencoders to decompose representations into quadratic polynomials, enabling analysis without input reference and describing structures from linear concepts to manifolds.
Sparse autoencoders are a standard tool for uncovering interpretable latent representations in neural networks. Yet, their interpretation depends on the inputs, making their isolated study incomplete. Polynomials offer a solution; they serve as algebraic primitives that can be analysed without reference to input and can describe structures ranging from linear concepts to complicated manifolds. This work uses bilinear autoencoders to efficiently decompose representations into quadratic polynomials. We discuss improvements that induce importance ordering, clustering, and activation sparsity. This is an initial step toward nonlinear yet analysable latents through their algebraic properties.