Balancing reconstruction error and Kullback-Leibler divergence in Variational Autoencoders
This addresses a delicate issue in VAE training for image generation, offering an incremental improvement over existing methods.
The paper tackles the problem of balancing reconstruction error and KL divergence in Variational Autoencoders, showing that replacing learned balancing with deterministic computation results in faster and more accurate performance, outperforming previous VAE architectures on datasets like Cifar and Celeba.
In the loss function of Variational Autoencoders there is a well known tension between two components: the reconstruction loss, improving the quality of the resulting images, and the Kullback-Leibler divergence, acting as a regularizer of the latent space. Correctly balancing these two components is a delicate issue, easily resulting in poor generative behaviours. In a recent work, Dai and Wipf obtained a sensible improvement by allowing the network to learn the balancing factor during training, according to a suitable loss function. In this article, we show that learning can be replaced by a simple deterministic computation, helping to understand the underlying mechanism, and resulting in a faster and more accurate behaviour. On typical datasets such as Cifar and Celeba, our technique sensibly outperforms all previous VAE architectures.