Cross-Modal Generative Augmentation for Visual Question Answering
This addresses the need for better data augmentation in multimodal learning, though it is incremental as it builds on existing generative frameworks.
The paper tackled the problem of data augmentation for multimodal models by introducing a generative model that creates samples of a target modality based on observed ones, improving UpDn-based models to achieve state-of-the-art performance in Visual Question Answering.
Data augmentation has been shown to effectively improve the performance of multimodal machine learning models. This paper introduces a generative model for data augmentation by leveraging the correlations among multiple modalities. Different from conventional data augmentation approaches that apply low-level operations with deterministic heuristics, our method learns a generator that generates samples of the target modality conditioned on observed modalities in the variational auto-encoder framework. Additionally, the proposed model is able to quantify the confidence of augmented data by its generative probability, and can be jointly optimised with a downstream task. Experiments on Visual Question Answering as downstream task demonstrate the effectiveness of the proposed generative model, which is able to improve strong UpDn-based models to achieve state-of-the-art performance.