Sparsity in Variational Autoencoders
This addresses model efficiency and generalization issues for researchers and practitioners using VAEs, but it is incremental as it discusses a known phenomenon.
The paper investigates the natural sparsity in Variational Autoencoders' latent spaces, known as overpruning, and argues it serves as self-regularization to reduce overfitting and guide model capacity tuning.
Working in high-dimensional latent spaces, the internal encoding of data in Variational Autoencoders becomes naturally sparse. We discuss this known but controversial phenomenon sometimes refereed to as overpruning, to emphasize the under-use of the model capacity. In fact, it is an important form of self-regularization, with all the typical benefits associated with sparsity: it forces the model to focus on the really important features, highly reducing the risk of overfitting. Especially, it is a major methodological guide for the correct tuning of the model capacity, progressively augmenting it to attain sparsity, or conversely reducing the dimension of the network removing links to zeroed out neurons. The degree of sparsity crucially depends on the network architecture: for instance, convolutional networks typically show less sparsity, likely due to the tighter relation of features to different spatial regions of the input.