Revisiting Structured Variational Autoencoders
This work addresses the practical implementation challenges of SVAEs for researchers and practitioners working with sequential data, offering incremental improvements through modern tools.
The paper tackled the difficulty of implementing structured variational autoencoders (SVAEs) by developing a modern implementation with hardware acceleration and parallelization, resulting in improved accuracy and efficiency over general alternatives, including better performance on prediction tasks and a novel self-supervised training approach for handling missing data.
Structured variational autoencoders (SVAEs) combine probabilistic graphical model priors on latent variables, deep neural networks to link latent variables to observed data, and structure-exploiting algorithms for approximate posterior inference. These models are particularly appealing for sequential data, where the prior can capture temporal dependencies. However, despite their conceptual elegance, SVAEs have proven difficult to implement, and more general approaches have been favored in practice. Here, we revisit SVAEs using modern machine learning tools and demonstrate their advantages over more general alternatives in terms of both accuracy and efficiency. First, we develop a modern implementation for hardware acceleration, parallelization, and automatic differentiation of the message passing algorithms at the core of the SVAE. Second, we show that by exploiting structure in the prior, the SVAE learns more accurate models and posterior distributions, which translate into improved performance on prediction tasks. Third, we show how the SVAE can naturally handle missing data, and we leverage this ability to develop a novel, self-supervised training approach. Altogether, these results show that the time is ripe to revisit structured variational autoencoders.