Latent Variables on Spheres for Autoencoders in High Dimensions
This addresses a fundamental bottleneck in generative modeling for complex data like images, offering a novel solution to improve inference precision without sacrificing sampling ability.
The paper tackles the dimensional dilemma in Variational Auto-Encoders (VAEs) by proposing a Spherical Auto-Encoder (SAE) that uses spherical normalization in the latent space, enabling high-dimensional latent codes for better reconstruction while maintaining stochastic sampling capabilities, with experiments validating its superiority.
Variational Auto-Encoder (VAE) has been widely applied as a fundamental generative model in machine learning. For complex samples like imagery objects or scenes, however, VAE suffers from the dimensional dilemma between reconstruction precision that needs high-dimensional latent codes and probabilistic inference that favors a low-dimensional latent space. By virtue of high-dimensional geometry, we propose a very simple algorithm, called Spherical Auto-Encoder (SAE), completely different from existing VAEs to address the issue. SAE is in essence the vanilla autoencoder with spherical normalization on the latent space. We analyze the unique characteristics of random variables on spheres in high dimensions and argue that random variables on spheres are agnostic to various prior distributions and data modes when the dimension is sufficiently high. Therefore, SAE can harness a high-dimensional latent space to improve the inference precision of latent codes while maintain the property of stochastic sampling from priors. The experiments on sampling and inference validate our theoretical analysis and the superiority of SAE.