Theoretical Convergence Guarantees for Variational Autoencoders
It addresses a theoretical gap for researchers and practitioners using VAEs, offering foundational insights but is incremental in extending existing optimization theory to VAEs.
This paper tackles the lack of theoretical convergence guarantees for Variational Autoencoders (VAEs) by providing non-asymptotic convergence rates of O(log n / sqrt(n)) for training with Stochastic Gradient Descent and Adam, applicable to various VAE variants.
Variational Autoencoders (VAE) are popular generative models used to sample from complex data distributions. Despite their empirical success in various machine learning tasks, significant gaps remain in understanding their theoretical properties, particularly regarding convergence guarantees. This paper aims to bridge that gap by providing non-asymptotic convergence guarantees for VAE trained using both Stochastic Gradient Descent and Adam algorithms.We derive a convergence rate of $\mathcal{O}(\log n / \sqrt{n})$, where $n$ is the number of iterations of the optimization algorithm, with explicit dependencies on the batch size, the number of variational samples, and other key hyperparameters. Our theoretical analysis applies to both Linear VAE and Deep Gaussian VAE, as well as several VAE variants, including $β$-VAE and IWAE. Additionally, we empirically illustrate the impact of hyperparameters on convergence, offering new insights into the theoretical understanding of VAE training.