Covariate-informed Representation Learning to Prevent Posterior Collapse of iVAE
This addresses a specific issue in representation learning for researchers using iVAEs, offering an incremental improvement to enhance model performance.
The paper tackled the posterior collapse problem in identifiable variational autoencoders (iVAEs), where latent representations become independent of observations given covariates, by proposing a covariate-informed iVAE (CI-iVAE) that prevents this collapse and results in latent representations containing more information from the observations.
The recently proposed identifiable variational autoencoder (iVAE) framework provides a promising approach for learning latent independent components (ICs). iVAEs use auxiliary covariates to build an identifiable generation structure from covariates to ICs to observations, and the posterior network approximates ICs given observations and covariates. Though the identifiability is appealing, we show that iVAEs could have local minimum solution where observations and the approximated ICs are independent given covariates.-a phenomenon we referred to as the posterior collapse problem of iVAEs. To overcome this problem, we develop a new approach, covariate-informed iVAE (CI-iVAE) by considering a mixture of encoder and posterior distributions in the objective function. In doing so, the objective function prevents the posterior collapse, resulting latent representations that contain more information of the observations. Furthermore, CI-iVAEs extend the original iVAE objective function to a larger class and finds the optimal one among them, thus having tighter evidence lower bounds than the original iVAE. Experiments on simulation datasets, EMNIST, Fashion-MNIST, and a large-scale brain imaging dataset demonstrate the effectiveness of our new method.