Integrating Random Effects in Variational Autoencoders for Dimensionality Reduction of Correlated Data
This addresses the limitation of standard VAEs for correlated datasets, which is common in real-world applications, but it is an incremental improvement by adapting existing statistical models to a deep learning framework.
The paper tackled the problem of correlated data in Variational Autoencoders (VAE) by proposing LMMVAE, which integrates random effects inspired by linear mixed models to handle correlations like spatial or temporal structures, resulting in significant improvements in squared reconstruction error and negative likelihood loss on unseen data and better performance in downstream tasks such as classification.
Variational Autoencoders (VAE) are widely used for dimensionality reduction of large-scale tabular and image datasets, under the assumption of independence between data observations. In practice, however, datasets are often correlated, with typical sources of correlation including spatial, temporal and clustering structures. Inspired by the literature on linear mixed models (LMM), we propose LMMVAE -- a novel model which separates the classic VAE latent model into fixed and random parts. While the fixed part assumes the latent variables are independent as usual, the random part consists of latent variables which are correlated between similar clusters in the data such as nearby locations or successive measurements. The classic VAE architecture and loss are modified accordingly. LMMVAE is shown to improve squared reconstruction error and negative likelihood loss significantly on unseen data, with simulated as well as real datasets from various applications and correlation scenarios. It also shows improvement in the performance of downstream tasks such as supervised classification on the learned representations.