Bootstrapping Disjoint Datasets for Multilingual Multimodal Representation Learning
This addresses a practical limitation in multilingual multimodal learning for AI applications, but it is incremental as it builds on existing methods to handle disjoint data.
The paper tackled the problem of learning multilingual multimodal representations from disjoint datasets where images don't overlap across languages, finding that aligned data performs better. They proposed a pseudopairing method to generate synthetic aligned triplets, improving retrieval performance without external data, though using external machine translation yielded better results.
Recent work has highlighted the advantage of jointly learning grounded sentence representations from multiple languages. However, the data used in these studies has been limited to an aligned scenario: the same images annotated with sentences in multiple languages. We focus on the more realistic disjoint scenario in which there is no overlap between the images in multilingual image--caption datasets. We confirm that training with aligned data results in better grounded sentence representations than training with disjoint data, as measured by image--sentence retrieval performance. In order to close this gap in performance, we propose a pseudopairing method to generate synthetically aligned English--German--image triplets from the disjoint sets. The method works by first training a model on the disjoint data, and then creating new triples across datasets using sentence similarity under the learned model. Experiments show that pseudopairs improve image--sentence retrieval performance compared to disjoint training, despite requiring no external data or models. However, we do find that using an external machine translation model to generate the synthetic data sets results in better performance.