Improving Bi-directional Generation between Different Modalities with Variational Autoencoders
This addresses a specific bottleneck in multimodal AI for researchers, offering incremental improvements to existing VAE methods.
The paper tackled the problem of joint representation collapse in multimodal variational autoencoders when generating large missing modalities, proposing JMVAE-kl and JMVAE-h models that prevent this issue and achieve equal or higher likelihood in bi-directional generation compared to conventional one-directional VAEs.
We investigate deep generative models that can exchange multiple modalities bi-directionally, e.g., generating images from corresponding texts and vice versa. A major approach to achieve this objective is to train a model that integrates all the information of different modalities into a joint representation and then to generate one modality from the corresponding other modality via this joint representation. We simply applied this approach to variational autoencoders (VAEs), which we call a joint multimodal variational autoencoder (JMVAE). However, we found that when this model attempts to generate a large dimensional modality missing at the input, the joint representation collapses and this modality cannot be generated successfully. Furthermore, we confirmed that this difficulty cannot be resolved even using a known solution. Therefore, in this study, we propose two models to prevent this difficulty: JMVAE-kl and JMVAE-h. Results of our experiments demonstrate that these methods can prevent the difficulty above and that they generate modalities bi-directionally with equal or higher likelihood than conventional VAE methods, which generate in only one direction. Moreover, we confirm that these methods can obtain the joint representation appropriately, so that they can generate various variations of modality by moving over the joint representation or changing the value of another modality.