Improving Multimodal Joint Variational Autoencoders through Normalizing Flows and Correlation Analysis
This work addresses the challenge of scalable and coherent multimodal data generation for applications in AI systems that process multiple data types, though it appears incremental as it builds on existing variational autoencoder and normalizing flow techniques.
The authors tackled the problem of generating coherent multimodal data by proposing a multimodal variational autoencoder that uses Deep Canonical Correlation Analysis embeddings and Normalizing Flows to improve joint and conditional generation. Their method demonstrated improvements in likelihood estimates, generation diversity, and coherence metrics across several datasets.
We propose a new multimodal variational autoencoder that enables to generate from the joint distribution and conditionally to any number of complex modalities. The unimodal posteriors are conditioned on the Deep Canonical Correlation Analysis embeddings which preserve the shared information across modalities leading to more coherent cross-modal generations. Furthermore, we use Normalizing Flows to enrich the unimodal posteriors and achieve more diverse data generation. Finally, we propose to use a Product of Experts for inferring one modality from several others which makes the model scalable to any number of modalities. We demonstrate that our method improves likelihood estimates, diversity of the generations and in particular coherence metrics in the conditional generations on several datasets.