Multimodal Generative Models for Scalable Weakly-Supervised Learning
This work addresses the challenge of scalable weakly-supervised learning across multiple modalities for AI systems, though it is incremental as it builds on existing generative approaches.
The paper tackles the problem of learning joint representations from multiple modalities with missing data by introducing a multimodal variational autoencoder (MVAE) that uses a product-of-experts inference network and sub-sampled training. It matches state-of-the-art performance on four datasets using fewer parameters and demonstrates applicability to weakly-supervised learning and robustness to incomplete supervision.
Multiple modalities often co-occur when describing natural phenomena. Learning a joint representation of these modalities should yield deeper and more useful representations. Previous generative approaches to multi-modal input either do not learn a joint distribution or require additional computation to handle missing data. Here, we introduce a multimodal variational autoencoder (MVAE) that uses a product-of-experts inference network and a sub-sampled training paradigm to solve the multi-modal inference problem. Notably, our model shares parameters to efficiently learn under any combination of missing modalities. We apply the MVAE on four datasets and match state-of-the-art performance using many fewer parameters. In addition, we show that the MVAE is directly applicable to weakly-supervised learning, and is robust to incomplete supervision. We then consider two case studies, one of learning image transformations---edge detection, colorization, segmentation---as a set of modalities, followed by one of machine translation between two languages. We find appealing results across this range of tasks.