Structure by Architecture: Structured Representations without Regularization
This addresses the need for better structured representations in generative modeling without relying on regularization, offering potential benefits for downstream tasks in machine learning.
The paper tackles the problem of self-supervised structured representation learning in autoencoders by proposing a sampling technique based on latent variable independence, avoiding the reconstruction-generative trade-off in VAEs, and demonstrates improved results in generation, disentanglement, and extrapolation on challenging image datasets.
We study the problem of self-supervised structured representation learning using autoencoders for downstream tasks such as generative modeling. Unlike most methods which rely on matching an arbitrary, relatively unstructured, prior distribution for sampling, we propose a sampling technique that relies solely on the independence of latent variables, thereby avoiding the trade-off between reconstruction quality and generative performance typically observed in VAEs. We design a novel autoencoder architecture capable of learning a structured representation without the need for aggressive regularization. Our structural decoders learn a hierarchy of latent variables, thereby ordering the information without any additional regularization or supervision. We demonstrate how these models learn a representation that improves results in a variety of downstream tasks including generation, disentanglement, and extrapolation using several challenging and natural image datasets.