Learning Multimodal Latent Space with EBM Prior and MCMC Inference
This work addresses multimodal generation tasks, which is incremental as it builds on existing methods by integrating EBM priors and MCMC inference.
The paper tackled the problem of multimodal generative modeling by proposing an approach that combines an energy-based model (EBM) prior with MCMC inference in the latent space, resulting in improved learning of shared latent variables for more coherent generation across modalities, as supported by empirical experiments.
Multimodal generative models are crucial for various applications. We propose an approach that combines an expressive energy-based model (EBM) prior with Markov Chain Monte Carlo (MCMC) inference in the latent space for multimodal generation. The EBM prior acts as an informative guide, while MCMC inference, specifically through short-run Langevin dynamics, brings the posterior distribution closer to its true form. This method not only provides an expressive prior to better capture the complexity of multimodality but also improves the learning of shared latent variables for more coherent generation across modalities. Our proposed method is supported by empirical experiments, underscoring the effectiveness of our EBM prior with MCMC inference in enhancing cross-modal and joint generative tasks in multimodal contexts.