Trading Information between Latents in Hierarchical Variational Autoencoders
This work provides incremental improvements for practitioners in machine learning by offering a theoretical and empirical framework to tune hierarchical VAEs for applications like representation learning and data compression.
The paper tackles the problem of optimizing hierarchical variational autoencoders (VAEs) by analyzing the rate/distortion trade-off across multiple latent layers, deriving theoretical bounds on downstream task performance and validating them with large-scale experiments to guide practitioners in selecting optimal rate configurations.
Variational Autoencoders (VAEs) were originally motivated (Kingma & Welling, 2014) as probabilistic generative models in which one performs approximate Bayesian inference. The proposal of $β$-VAEs (Higgins et al., 2017) breaks this interpretation and generalizes VAEs to application domains beyond generative modeling (e.g., representation learning, clustering, or lossy data compression) by introducing an objective function that allows practitioners to trade off between the information content ("bit rate") of the latent representation and the distortion of reconstructed data (Alemi et al., 2018). In this paper, we reconsider this rate/distortion trade-off in the context of hierarchical VAEs, i.e., VAEs with more than one layer of latent variables. We identify a general class of inference models for which one can split the rate into contributions from each layer, which can then be tuned independently. We derive theoretical bounds on the performance of downstream tasks as functions of the individual layers' rates and verify our theoretical findings in large-scale experiments. Our results provide guidance for practitioners on which region in rate-space to target for a given application.